[00:32] *** swebb has joined #internetarchive.bak [00:43] https://code.facebook.com/posts/1433093613662262/-under-the-hood-facebook-s-cold-storage-system-/ [00:43] hardcore "we updated the firmware in the drive controller to enforce this constraint" [01:08] 03registrar 05master d96f0df 06other 10SHARD6/pubkeys registration of sunny on SHARD6 [01:17] *** GLaDOS has quit IRC (Read error: Operation timed out) [01:33] *** GLaDOS has joined #internetarchive.bak [01:33] *** svchfoo1 sets mode: +o GLaDOS [01:44] *** primus104 has quit IRC (Leaving.) [01:49] *** Start has joined #internetarchive.bak [04:37] *** zottelbey has joined #internetarchive.bak [05:55] *** kalleboo has joined #internetarchive.bak [07:11] *** Vito` has joined #internetarchive.bak [07:12] *** Vito` has left [07:54] *** kalleboo has quit IRC (Leaving...) [08:05] db48x: 1. i applied your patch; 2. where's the typo? [08:06] *** primus104 has joined #internetarchive.bak [08:16] *** kalleboo has joined #internetarchive.bak [08:21] protodev: s/remove/remote/ [08:22] oh [08:22] it put the pull request for the typo fix into my own fork :P [08:22] https://github.com/db48x/IA.BAK.Perl/pull/1 [08:22] very dumb [08:23] 03registrar 05master cca52bc 06other 10SHARD4/pubkeys registration of protodev on SHARD4 [08:23] U SERIOUS !?!!! [08:23] * protodev raging [08:23] now it works, thx [08:23] yea :) [08:23] you're welcome [08:34] *** atomotic has joined #internetarchive.bak [08:38] 03registrar 05master 161c745 06other 10SHARD3/pubkeys registration of protodev on SHARD3 [09:02] closure: I had a play and git annex as shipped currently is happy to start simultaneous gets of two 100GB files and a 30GB, a 35GB and a 40GB file onto a 250GB disk; so I think that's definitely a bug :) [09:03] heh [09:05] Senji: so picky of you ;) [09:14] *** kalleboo has quit IRC (Leaving...) [10:02] *** balrog has quit IRC (Read error: Operation timed out) [10:03] *** Lord_Nigh has quit IRC (Write error: Broken pipe) [10:03] *** atomotic has quit IRC (Quit: Textual IRC Client: www.textualapp.com) [10:05] *** logchfoo_ starts logging #internetarchive.bak at Fri May 08 10:05:15 2015 [10:05] *** logchfoo_ has joined #internetarchive.bak [10:06] *** GLaDOS has quit IRC (Read error: Operation timed out) [10:07] *** GLaDOS has joined #internetarchive.bak [10:07] *** svchfoo2 sets mode: +o GLaDOS [10:08] *** Ctrl-S has quit IRC (Read error: Operation timed out) [10:10] replacement disk is DOA :( [10:24] *** marvinw has joined #internetarchive.bak [10:27] *** primus104 has quit IRC (Leaving.) [10:29] *** balrog has joined #internetarchive.bak [10:37] *** atomotic has joined #internetarchive.bak [10:43] *** GLaDOS has quit IRC (Ping timeout: 252 seconds) [10:43] *** GLaDOS has joined #internetarchive.bak [10:44] *** svchfoo2 sets mode: +o GLaDOS [11:01] *** S[h]O[r]T has joined #internetarchive.bak [11:02] *** sep332 has joined #internetarchive.bak [11:02] *** svchfoo2 sets mode: +o sep332 [11:03] *** Ctrl-S has joined #internetarchive.bak [11:07] *** atomotic has quit IRC (Quit: My Mac has gone to sleep. ZZZzzz…) [11:10] *** toad1 has joined #internetarchive.bak [11:17] *** GLaDOS has quit IRC (Ping timeout: 252 seconds) [11:17] *** GLaDOS has joined #internetarchive.bak [11:18] *** svchfoo2 sets mode: +o GLaDOS [11:19] *** kalleboo has joined #internetarchive.bak [11:50] thats...handy [11:51] *** db48x has quit IRC (Ping timeout: 258 seconds) [11:58] *** VADemon has joined #internetarchive.bak [12:40] *** sankin has joined #internetarchive.bak [13:12] 03registrar 05master c4626d3 06other 10SHARD3/pubkeys registration of protodev on SHARD3 [13:14] 03registrar 05master 64a1930 06other 10SHARD4/pubkeys registration of protodev on SHARD4 [13:18] *** kalleboo has quit IRC (Quit: Leaving...) [13:22] Senji: ah, yes, simulantaneous downloads will indeed bypass the size checking [13:23] you could certianly file a bug about that. (I can't guarantee a speedy fix) [13:23] I was only doing a single download for the earlier report, btw, because I forsaw that multiple might be a problem at the end [13:26] hmm, it does have files on disk it could check, so maybe a fix is somewhat possible. may be hard due to concurrency issues tho [13:30] oho, it's even got locking to know which are still in progress [13:30] * closure downgrades estimate to half an hour to fix [13:30] Heh [13:31] Do you want me to formally file a bug; and if so where? [13:31] getTransfers >>= fmap sumsizes >>= checkdiskspace [13:31] Although I'm about to crash out after allnighting the election, so probably shouldn't try filing bugs until I'm properly awake again [13:33] I'll probably remember this one, it's a pretty glaring hole once brought to attention. but, git-annex.branchable.com/bugs [14:25] *** trs80 has quit IRC (Remote host closed the connection) [14:25] *** primus104 has joined #internetarchive.bak [14:36] *** londoncal has joined #internetarchive.bak [14:51] *** trs80 has joined #internetarchive.bak [15:55] *** sankin has quit IRC (Leaving.) [16:02] *** GLaDOS has quit IRC (Read error: Operation timed out) [16:05] *** GLaDOS has joined #internetarchive.bak [16:05] *** svchfoo2 sets mode: +o GLaDOS [16:29] *** trs80 has quit IRC (Remote host closed the connection) [16:40] *** GLaDOS has quit IRC (Ping timeout: 252 seconds) [16:43] *** GLaDOS has joined #internetarchive.bak [16:43] *** svchfoo3 sets mode: +o GLaDOS [17:01] *** Start has quit IRC (ircd.choopa.net ircd.shaw.ca) [17:01] *** DFJustin has quit IRC (ircd.choopa.net ircd.shaw.ca) [17:01] *** SketchCow has quit IRC (ircd.choopa.net ircd.shaw.ca) [17:01] *** csssuf has quit IRC (ircd.choopa.net ircd.shaw.ca) [17:01] *** chfoo- has quit IRC (ircd.choopa.net ircd.shaw.ca) [17:01] *** matthusby has quit IRC (ircd.choopa.net ircd.shaw.ca) [17:01] *** underscor has quit IRC (ircd.choopa.net ircd.shaw.ca) [17:05] *** Start has joined #internetarchive.bak [17:05] *** DFJustin has joined #internetarchive.bak [17:05] *** SketchCow has joined #internetarchive.bak [17:05] *** csssuf has joined #internetarchive.bak [17:05] *** underscor has joined #internetarchive.bak [17:05] *** matthusby has joined #internetarchive.bak [17:05] *** chfoo- has joined #internetarchive.bak [17:05] *** ircd.shaw.ca sets mode: +oo DFJustin SketchCow [17:28] *** Start has quit IRC (Disconnected.) [17:29] I am not seeing a lot of growth in the saved data lately. [17:29] *** Start has joined #internetarchive.bak [17:30] *** Start has quit IRC (Client Quit) [17:31] *** londoncal has quit IRC (Remote host closed the connection) [17:31] *** GLaDOS has quit IRC (Read error: Operation timed out) [17:32] *** GLaDOS has joined #internetarchive.bak [17:33] *** primus104 has quit IRC (Leaving.) [17:33] *** svchfoo2 sets mode: +o GLaDOS [17:41] *** db48x has joined #internetarchive.bak [17:42] *** svchfoo2 sets mode: +o db48x [17:42] closure: thanks for fixing that windows bug in git-annex :) [17:43] I assume disk space download is reported at the end of a save? Or elsewhere? [17:43] There's a good amount of people who signed on in the last few days [17:44] SketchCow: hourly by default, though it's a bit fragile [17:44] Just seeing. [17:44] OK. [17:44] if the user has set up a cron job then it'll report daily or monthly or whatever [17:45] So the reporting can be way out of date of how much of each shard is there? [17:45] if we integrate the cronjob branch (closure, I sent you a pull request; could you look over the patch?) then that won't be a manual step [17:45] yea [17:45] can be out of sync in the other direction as well [17:45] my computer celebdil lost a drive, so it's stopped reporting for that copy and will expire in a week or so [17:47] db48x: can you confirm the fix? [17:47] (to the windows bug) [17:48] closure: sadly no; I had to file another, different bug report [17:48] when it launches ssh the first time (to verify that it can connect to the host), it fails with an error message stating that libcrypto-1.0.0.dll is missing :P [17:49] sounds like the old version of ssh [17:49] I uninstalled git-annex and git and reinstalled them both; same result [17:51] iabak instructs the user to make a cronjob. If the user sticks to the instructions or copy& pastes, it will update once per week [17:51] on mondays [17:52] db48x: i thought we were expiring them after a month, not a week? [17:52] I was guessing [17:53] we are expiring after a month, but a week would be better, and the cron job needs to run more frequently [17:54] on the cronjob branch I set it to daily [17:58] db48x: thanks, I'll have to write some horrible batch script to get ssh in path correctly [17:58] well, you're welcome [17:58] windows should come with eye-poking instuments, just for break time [17:58] yes! [17:58] maybe a home waterboarding kit [18:00] Microsoft Hydrosurface [18:01] closure: maybe just run it by absolute path instead? [18:01] oh, no. it's git that's running it [18:03] week vs month for expiry: if we consider the storage 'nearly-online', wouldn't a week be too quickly? Running the cron once a week seems like a sweet spot to me [18:05] *** GLaDOS has quit IRC (Ping timeout: 252 seconds) [18:07] run once a month, and then the user's computer is off that day [18:07] the cron job should take about 10 minutes to run, [18:08] *** GLaDOS has joined #internetarchive.bak [18:08] *** svchfoo2 sets mode: +o GLaDOS [18:12] why is expiring more frequently helpful? [18:12] *** ersi has left [18:12] db48x: reviewing your patch, what about systems that have systemd installed, but the user doesn't log into in a way that starts the user-level systemd? [18:14] iirc, console logins or ssh logins might not. I'm not sure about that. Seems they'd have to be logged in for it to run in any case. [18:17] *** GLaDOS has quit IRC (Read error: Operation timed out) [18:17] I think that 2>&- may be a bashism [18:18] *** GLaDOS has joined #internetarchive.bak [18:18] hmm, seems not [18:18] *** svchfoo2 sets mode: +o GLaDOS [18:29] *** GLaDOS has quit IRC (Ping timeout: 252 seconds) [18:30] *** GLaDOS has joined #internetarchive.bak [18:30] *** svchfoo3 sets mode: +o GLaDOS [18:41] *** VADemon_ has joined #internetarchive.bak [18:42] *** VADemon has quit IRC (Ping timeout: 512 seconds) [18:58] *** primus104 has joined #internetarchive.bak [19:33] *** Start has joined #internetarchive.bak [19:41] https://archive.org/details/git-annex-builds [19:41] * closure should have done this a while ago. Nicely recursive too ;) [19:41] * closure uploads 6 gb worth of past builds [19:47] this is really nice cause recent git-annex can download right from archive.org over the web after uploading to it over S3 :) [20:10] * closure watches the derive stall [20:22] *** Start has quit IRC (Disconnected.) [20:37] *** Start has joined #internetarchive.bak [20:54] *** db48x` has joined #internetarchive.bak [21:20] *** Start has quit IRC (Disconnected.) [22:09] *** db48x` has quit IRC (Read error: Operation timed out) [23:22] *** db48x` has joined #internetarchive.bak [23:31] *** db48x` has quit IRC (Read error: Operation timed out)