[00:07] bah [00:12] *** kyan has joined #internetarchive.bak [00:24] ok, so.. only 5 of the repos in shard1 have run iabak lately, it seems. the other 30-some could all be deleted for all I know [00:25] and I'll bet some of them are, there are 7 different repos that all claim to be at root@katie:~/src/IA.BAK/shard1 [00:25] http://iabak.archiveteam.org/stats/SHARD1.expire [00:25] if you see yoursef in this list, you're not reporting in and are a candidate for expire [00:28] Nice on the shard progress [00:28] closure: I meant to ask you about that one; I've run fsck in that repo and synced, but I'm still in there [00:29] which repo is it? [00:29] db48x@celebdil [00:29] does it have the same annex.uuid as is listed there? [00:30] closure: Sounds like we need to start getting some sort of mailing list/registration going. [00:30] I'd prefer not to start expiring repos just yet. [00:30] SketchCow: agreed, not sure where to store the emails though (not in git I know that) [00:30] I'd like us to go through the logs in this channel, and we can find which names added things. [00:31] yes [00:32] db48x: hmm. sure you've fscked since upgrading to a new enough git-annex? [00:32] probably? dunno [00:32] I'll rerun it [00:33] closure: Mailing list? [00:33] activity.log | 16 +--------------- [00:33] woah! [00:34] someone ended up committing a change that nuked 16 fscks out of the activity log [00:34] that's really strange [00:35] heh [00:35] I'm guessing that means this new feature has a bug.. [00:35] sounds like it [00:42] * closure reproduced it [00:43] great thing about working with git based stuff, so much info and reproducability.. [00:46] that's a good one to find [00:48] ooh, interesting [00:48] http://hastebin.com/raw/zifuhiquja [01:02] ok, that expiry is fixed, but I've already done a release today, so it'll be a while [01:03] maybe I'll switch iabak to the git-annex daily builds for a while [01:11] db48x: well spotted, fixing that too, although it won't cause any problems [01:46] closure: weird that 09fde827-7a62-42c7-ab92-f7442e7fb289 ubuntu@iabak:~/IA.BAK/shard1 is on the list [01:46] AFAIK that one is still syncing [01:47] do I have to do something else to get it to report in? [02:10] yipdw: turns out the code was broken. So, will need to wait until there's a new git-annex to use [02:24] *** wp494 has quit IRC (Quit: LOUD UNNECESSARY QUIT MESSAGES) [02:24] *** wp494 has joined #internetarchive.bak [03:01] *** ppiixx has quit IRC (Remote host closed the connection) [03:02] *** ppiixx has joined #internetarchive.bak [03:15] *** Please restart your iabak's *** [03:16] or just run install-git-annex by hand [03:24] closure: ah ok [03:33] *** niyaje4 has quit IRC (Ping timeout: 600 seconds) [03:55] *** niyaje4 has joined #internetarchive.bak [04:36] *** chfoo- has joined #internetarchive.bak [04:42] *** chfoo- has quit IRC (Quit: ZNC - 1.6.0 - http://znc.in) [04:45] *** chfoo- has joined #internetarchive.bak [04:51] *** wp494 has quit IRC (ircd.choopa.net ircd.shaw.ca) [04:51] *** Start has quit IRC (ircd.choopa.net ircd.shaw.ca) [04:51] *** matthusb- has quit IRC (ircd.choopa.net ircd.shaw.ca) [04:51] *** Sanqui has quit IRC (ircd.choopa.net ircd.shaw.ca) [04:51] *** underscor has quit IRC (ircd.choopa.net ircd.shaw.ca) [04:51] *** aschmitz has quit IRC (ircd.choopa.net ircd.shaw.ca) [04:51] *** csssuf has quit IRC (ircd.choopa.net ircd.shaw.ca) [04:51] *** DFJustin has quit IRC (ircd.choopa.net ircd.shaw.ca) [04:51] *** pikhq has quit IRC (ircd.choopa.net ircd.shaw.ca) [04:51] *** achip has quit IRC (Read error: Operation timed out) [04:51] *** SN4T14__ has quit IRC (ircd.choopa.net ny.us.hub) [04:51] *** chfoo has quit IRC (ircd.choopa.net ny.us.hub) [04:51] *** espes___ has quit IRC (ircd.choopa.net ny.us.hub) [04:51] *** Quile has quit IRC (ircd.choopa.net ny.us.hub) [04:51] *** SketchCow has quit IRC (ircd.choopa.net ny.us.hub) [04:51] *** closure has quit IRC (ircd.choopa.net ny.us.hub) [04:51] *** ersi_ has quit IRC (ircd.choopa.net ny.us.hub) [04:51] *** bpye_ has quit IRC (ircd.choopa.net ny.us.hub) [04:51] *** serapeum has quit IRC (ircd.choopa.net ny.us.hub) [04:51] *** svchfoo3 has quit IRC (ircd.choopa.net ny.us.hub) [04:51] *** destrudo has quit IRC (ircd.choopa.net ny.us.hub) [04:51] *** Cameron_D has quit IRC (ircd.choopa.net ny.us.hub) [04:51] *** ppiixx has quit IRC (ircd.choopa.net ny.us.hub) [04:51] *** hatseflat has quit IRC (ircd.choopa.net ny.us.hub) [04:51] *** balrog has quit IRC (ircd.choopa.net ny.us.hub) [04:51] *** Muad-Dib has quit IRC (ircd.choopa.net ny.us.hub) [04:51] *** lhobas has quit IRC (ircd.choopa.net ny.us.hub) [04:51] *** mrfoo has quit IRC (ircd.choopa.net ny.us.hub) [04:51] *** jbenet_ has quit IRC (ircd.choopa.net ny.us.hub) [04:51] *** ryang has quit IRC (ircd.choopa.net ny.us.hub) [04:51] *** antomatic has quit IRC (ircd.choopa.net ny.us.hub) [04:51] *** Kazzy has quit IRC (ircd.choopa.net ny.us.hub) [04:51] *** edsu_ has quit IRC (ircd.choopa.net ny.us.hub) [04:51] *** chfoo- has quit IRC (ircd.choopa.net ny.us.hub) [04:51] *** hater has quit IRC (ircd.choopa.net ny.us.hub) [04:51] *** Senji has quit IRC (ircd.choopa.net ny.us.hub) [04:51] *** garyrh has quit IRC (ircd.choopa.net ny.us.hub) [04:51] *** raylee has quit IRC (ircd.choopa.net ny.us.hub) [04:51] *** svchfoo2 has quit IRC (ircd.choopa.net ny.us.hub) [04:51] *** Atluxity has quit IRC (ircd.choopa.net ny.us.hub) [04:54] *** wp494 has joined #internetarchive.bak [04:54] *** Start has joined #internetarchive.bak [04:54] *** matthusb- has joined #internetarchive.bak [04:54] *** pikhq has joined #internetarchive.bak [04:54] *** DFJustin has joined #internetarchive.bak [04:54] *** csssuf has joined #internetarchive.bak [04:54] *** aschmitz has joined #internetarchive.bak [04:54] *** underscor has joined #internetarchive.bak [04:54] *** Sanqui has joined #internetarchive.bak [04:54] *** ircd.shaw.ca sets mode: +ooo Start pikhq DFJustin [05:15] *** niyaje4 has quit IRC (Ping timeout: 601 seconds) [07:10] *** zottelbey has joined #internetarchive.bak [07:10] *** chfoo- has joined #internetarchive.bak [07:10] *** ppiixx has joined #internetarchive.bak [07:10] *** raylee has joined #internetarchive.bak [07:10] *** SN4T14__ has joined #internetarchive.bak [07:10] *** svchfoo2 has joined #internetarchive.bak [07:10] *** Senji has joined #internetarchive.bak [07:10] *** ersi_ has joined #internetarchive.bak [07:10] *** hatseflat has joined #internetarchive.bak [07:10] *** balrog has joined #internetarchive.bak [07:10] *** Muad-Dib has joined #internetarchive.bak [07:10] *** edsu_ has joined #internetarchive.bak [07:10] *** jbenet_ has joined #internetarchive.bak [07:10] *** mrfoo has joined #internetarchive.bak [07:10] *** lhobas has joined #internetarchive.bak [07:10] *** bpye_ has joined #internetarchive.bak [07:10] *** ryang has joined #internetarchive.bak [07:10] *** serapeum has joined #internetarchive.bak [07:10] *** Atluxity has joined #internetarchive.bak [07:10] *** chfoo has joined #internetarchive.bak [07:10] *** antomatic has joined #internetarchive.bak [07:10] *** svchfoo3 has joined #internetarchive.bak [07:10] *** destrudo has joined #internetarchive.bak [07:10] *** Cameron_D has joined #internetarchive.bak [07:10] *** espes___ has joined #internetarchive.bak [07:10] *** closure has joined #internetarchive.bak [07:10] *** ny.us.hub sets mode: +oooo svchfoo2 chfoo svchfoo3 closure [07:10] *** SketchCow has joined #internetarchive.bak [07:10] *** Quile has joined #internetarchive.bak [07:10] *** hater has joined #internetarchive.bak [07:10] *** Kazzy has joined #internetarchive.bak [07:10] *** ny.us.hub sets mode: +oo SketchCow Kazzy [07:14] http://iabackup.archiveteam.org/ia.bak/ is coming along. I hope we have the space around! [07:15] realeyes: Thanks again for stepping in. [07:17] *** garyrh has joined #internetarchive.bak [07:17] *** svchfoo1 sets mode: +o garyrh [07:35] *** atomotic has joined #internetarchive.bak [08:42] oops, filled up my root disk :P [09:03] Awwww yis [09:09] *** ersi_ is now known as ersi [09:10] *** svchfoo2 sets mode: +o ersi [09:30] damn latency, got it up to 1500KB/s with 6 sessions but my line is still mostly idle [09:30] 78GB backed up here so far [09:51] ppiixx: :) [09:51] I'm lucky when mine hits 1200 [09:51] someone patch in multithreaded wget to git annex ;) [09:54] closure's working on letting git annex start multiple transfers at once [09:55] cool [09:55] I see an uptick in the graph! Hurrah [09:56] SketchCow: yea! [10:52] *** atomotic has quit IRC (Quit: Textual IRC Client: www.textualapp.com) [11:44] *** dirt has quit IRC (Read error: Operation timed out) [11:44] *** Lord_Nigh has quit IRC (Read error: Operation timed out) [11:44] *** arkiver has quit (Read error: Operation timed out) [11:45] *** logchfoo has quit (Read error: Operation timed out) [11:45] *** phuzion has quit (Read error: Operation timed out) [11:45] *** sep332 has quit (Read error: Operation timed out) [11:45] *** dirt (james@[redacted]) has joined #internetarchive.bak [11:46] *** Lord_Nigh (~Lord_Nigh@[redacted]) has joined #internetarchive.bak [11:46] *** GLaDOS has quit (Read error: Operation timed out) [11:47] *** richo has quit (Read error: Operation timed out) [11:47] *** S[h]O[r]T has quit (Read error: Operation timed out) [11:47] *** marvinw has quit (Ping timeout: 600 seconds) [11:48] *** arkiver (~arkiver@[redacted]) has joined #internetarchive.bak [11:48] *** svchfoo3 gives channel operator status to arkiver [11:48] *** toad1 has quit (Read error: Operation timed out) [11:49] *** GLaDOS (~STR_IDENT@[redacted]) has joined #internetarchive.bak [11:49] *** svchfoo2 gives channel operator status to GLaDOS [11:49] *** Ctrl-S has quit (Read error: Operation timed out) [11:58] *** S[h]O[r]T (~Sh]Or]T@[redacted]) has joined #internetarchive.bak [11:58] *** phuzion (~phuzion@[redacted]) has joined #internetarchive.bak [11:58] *** richo (~richo@[redacted]) has joined #internetarchive.bak [11:59] *** marvinw (~marvinw@[redacted]) has joined #internetarchive.bak [11:59] *** atomotic (~atomotic@[redacted]) has joined #internetarchive.bak [12:00] *** sep332 (~sep332@[redacted]) has joined #internetarchive.bak [12:00] *** svchfoo3 gives channel operator status to sep332 [12:01] *** Ctrl-S (~Ctrl-S@[redacted]) has joined #internetarchive.bak [12:03] *** toad1 (~toad@[redacted]) has joined #internetarchive.bak [12:50] *** Start has quit (Remote host closed the connection) [12:50] *** Start (~Start@[redacted]) has joined #internetarchive.bak [12:50] *** svchfoo2 gives channel operator status to Start [12:57] *** Start has quit (Read error: Connection reset by peer) [12:57] *** Start_ (~Start@[redacted]) has joined #internetarchive.bak [12:58] *** sankin (~sankin@[redacted]) has joined #internetarchive.bak [13:24] *** Start_ has quit (Read error: Connection reset by peer) [13:36] *** Start (~Start@[redacted]) has joined #internetarchive.bak [13:36] *** svchfoo2 gives channel operator status to Start [13:48] *** Start has quit (Disconnected.) [13:55] *** achip (~thechip@[redacted]) has joined #internetarchive.bak [14:33] *** Start (~Start@[redacted]) has joined #internetarchive.bak [14:34] *** Start has quit (Read error: Connection reset by peer) [14:34] *** Start_ (~Start@[redacted]) has joined #internetarchive.bak [14:35] *** toad1 (~toad@[redacted]) has left #internetarchive.bak [14:38] *** atomotic has quit (Quit: Textual IRC Client: www.textualapp.com) [15:52] *** Start_ has quit (Disconnected.) [15:59] *** Vito` (sid7616@[redacted]) has joined #internetarchive.bak [16:12] *** Start (~Start@[redacted]) has joined #internetarchive.bak [16:14] *** Start has quit (Read error: Connection reset by peer) [16:14] *** Start_ (~Start@[redacted]) has joined #internetarchive.bak [16:53] *** Start_ has quit (Ping timeout: 370 seconds) [16:58] *** Start-mob (~Start-mob@[redacted]) has joined #internetarchive.bak [17:03] *** Start-mob has quit (Remote host closed the connection) [17:47] *** atomotic (~atomotic@[redacted]) has joined #internetarchive.bak [17:51] *** Start (~Start@[redacted]) has joined #internetarchive.bak [17:52] *** Start has quit (Read error: Connection reset by peer) [17:52] *** Start_ (~Start@[redacted]) has joined #internetarchive.bak [18:27] *** atomotic has quit (Quit: My Mac has gone to sleep. ZZZzzz…) [18:33] *** Start_ has quit (Disconnected.) [19:02] *** Start-mob (~Start-mob@[redacted]) has joined #internetarchive.bak [19:05] *** Start-mob has quit (Remote host closed the connection) [19:29] btw i started rewriting IA.BAK in perl [19:31] *** Start (~Start@[redacted]) has joined #internetarchive.bak [19:32] *** Start has quit (Read error: Connection reset by peer) [19:32] *** Start (~Start@[redacted]) has joined #internetarchive.bak [19:33] what's the advantage? [19:37] it'll work on *bsd and osx [19:37] without having to rewrite the bash scripts in a posix-compliant way [19:39] we're already having users download git and git-annex. we could add bash too lol [19:41] sep332: no problem: git and git-annex remain the same [19:42] i mean have the script download bash binaries and run the rest in bash [19:43] osx ships with bash, doesn't putting #!/bin/bash at the top make it execute with bash? [19:43] Vito`: the problem is not bash itself, but the coreutils [19:43] f.e: osx does not support find -printf [19:43] ah [19:44] and so on [19:44] there is a huge list of missing features [19:46] some guys on freenode #bash showed me the posix list of minimal subset of supported features and reading the list will take longer than rewriting the script [19:48] heh [19:49] *** SN4T14_ (~SN4T14@[redacted]) has joined #internetarchive.bak [19:51] So, little note. [19:51] Assumptively, X amount of people can code in bash. [19:51] Y people can code in prl. [19:51] Smashing a component of a shared project from one language to another.... that's not a light thing. [19:51] If Y is less than X [19:53] *** SN4T14__ has quit (Read error: Operation timed out) [19:53] i understand your point, but atm there are more languages in use: awk, perl, bash and so on [19:58] *** sankin has quit (Leaving.) [20:02] In the code we're using? [20:03] Let's put it this way. Put it by closure. If he's all for it, go for it. [20:03] *** Start has quit (Disconnected.) [20:03] Otherwise, it should be a IA.BAK-osx script [20:03] closure and db48x are doing the lion's share of hacking this thing [20:07] SketchCow: @closure | well, if you do it in perl, I can help maintain it [20:08] that's what he said yesterday [20:22] OK, concern withdrawn, proceed. [20:23] coo, what's the huge bulge in 4 copies this afternoon? [20:24] folks downloading stuff [20:24] very very quickly [20:25] There's a couple of nearly vertical lines in that progress graph [20:48] how often do clients time out from the graphs? [20:48] I haven't been downloading from 03868 (up in New Hampshire) for days but it's still on the map [20:52] *** Start-mob (~Start-mob@[redacted]) has joined #internetarchive.bak [20:59] *** Start-mob has quit (Remote host closed the connection) [21:20] *** Start-mob (~Start-mob@[redacted]) has joined #internetarchive.bak [21:26] *** Start-mob has quit (Ping timeout: 370 seconds) [21:31] db48x: http://hastebin.com/gomekohacu.vbs i'm porting iabak-helper and i seriously have no idea what this line is doing? [21:32] what part of it is unclear? [21:33] git annex find prints out a bunch of filenames [21:33] we pipe that to dirname to get the names of the directories that they're in [21:33] uniq eliminates the duplicates, giving us just the unique directory names [21:34] shuf shuffles the list [21:34] git annex get downloads each item in the list [21:34] ok, thx [21:38] is it faster to rewrite some parts of this line in perl or should i just call system("...")? [21:39] Eww. I'd try and avoid system() as much as possible. Also OSX might not have shuf. [21:39] Senji: i'm seriously NOT going to rewrite git-annex [21:40] no shut on yosemite [21:40] *shuf [21:40] oh, ok [21:42] hater: well, no. :-). Executing complex pipelines in system is a recipe for fragility though. [21:42] hater: Well, some parts of git-annex are probably quite sensible to reimplement. :) [21:42] ... not much used here mind you. [21:44] one possibility for shuf on OS X is to also try gshuf; if someone's doing this it's not totally out of the question that they'll also have eg homebrew installed [21:45] yipdw: shuffling a list in perl is no problem [21:45] sure [21:46] that said the g* pattern has worked before when we didn't want to bother reimplementing coreutils [21:46] sure it's not in the base install but meh [21:55] * closure has concurrent git-annex get working. Not quite ready for general public, but it's gonna be nice! [21:56] Nomnom [22:06] mmh, dropped 600GB into here overnight, now to see if i can make my home connection pull from IA at anything faster than 300KB/s [22:24] *** Start (~Start@[redacted]) has joined #internetarchive.bak [22:24] *** svchfoo3 gives channel operator status to Start [22:36] Kazzy: very nice :) [22:40] *** zottelbey has quit (Remote host closed the connection) [23:10] *** logchfoo_ starts logging #internetarchive.bak at Fri Apr 10 23:10:10 2015 [23:10] *** logchfoo_ has joined #internetarchive.bak [23:14] *** logchfoo starts logging #internetarchive.bak at Fri Apr 10 23:14:25 2015 [23:14] *** logchfoo has joined #internetarchive.bak [23:32] *** niyaje4 has joined #internetarchive.bak