[01:18] *** yipdw_ is now known as yipdw [01:19] *** svchfoo1 sets mode: +o yipdw [01:44] *** iten has joined #internetarchive.bak [01:48] can I sign up for SHARD2@iabak.archiveteam.org:shard2? [01:50] Graphs broke [01:51] iten: of course! pastbin your key and I'll add it [01:51] SketchCow: could be my fault, or maybe nobody's connected [01:51] ok, cool [01:52] wait, pastebin or "paste in"? [01:52] pastebin is best [01:55] ok, http://pastebin.com/sMqrbDMu [01:55] No, there's a problem [01:55] But closure will recognize [01:58] SketchCow: yes, I was being slightly facetious :) [02:00] closure: i'm still getting one failure for shard1 [02:02] oh. interesting. https://archive.org/download/Ttscribe/Ttscribe_files.xml has been darked [02:05] so i'm complete and fscked except for that file [02:05] what's the failure? [02:06] 403 [02:06] ah [02:06] we'll just remove it from the collection [02:06] why do they dark things? [02:06] (and when will that be done) [02:06] like are we gonna have problems having things they don't [02:06] for these items, it could be a dmca request or similar [02:07] yea, we'll end up with some files that were available for a while and then went dark, but that's not really a problem [02:07] ok [02:07] well, not for us anyway [02:07] when will that be removed so i can mark this as complete? [02:08] if we were backing up wayback machine stuff, then it could be a site that changed their robots.txt [02:12] ima hold off the fsck until that is removed [02:12] no worries [02:31] thanks! the server still isn't accepting my key though, should I just wait till it updates or something, or did I screw something up? [02:31] *** beardicus has quit IRC (My MacBook Pro has gone to sleep. ZZZzzz…) [02:37] hrm! you're not the first to report that [02:37] what's your ip address? [02:39] 24.4.230.103 [02:42] Apr 07 22:30:28 ia-bak sshd[16070]: Connection closed by 24.4.230.103 [preauth] [02:44] no other details though [02:45] yeah, all I get is 'Permission denied (publickey,password).' [02:47] just to double check, the key you gave me is the key that iabak generated for you? [02:47] it's stored in the id_rsa file in the same directory as iabak [02:47] maybe tmux screwed up the copy/paste, let me double-check [02:49] seems to be exactly the same (as in id_rsa.pub that is) [02:50] the line previous to mine in the "pubkeys" file you committed starts with "sh-rsa" rather than "ssh-rsa" [02:50] if that matters [02:51] that does matter :) [02:51] heh, indeed [02:51] hater: that one's yours, so maybe that's the problem :) [02:54] :) [03:00] *** toad1 has joined #internetarchive.bak [03:37] *** chfoo has quit IRC (Remote host closed the connection) [03:40] *** chfoo has joined #internetarchive.bak [03:41] *** svchfoo3 sets mode: +o chfoo [04:28] http://iabak.archiveteam.org:8080/render/?width=1060&height=731&_salt=1428467312.185&from=00%3A00_20150407&until=23%3A59_20150407&lineMode=connected&tz=UTC&logBase=&bgcolor=000000&majorGridLineColor=FFFFFF&fgcolor=FFFFFF&minorGridLineColor=C0C0C0&colorList=red%2Cgold%2Cgreen&lineWidth=2&target=alias%28iabak.shardstats.numcopies.0.shard2%2C%220%20copies%22%29&target=alias%28sumSeries%28iabak.shardstats.numcopies.1.shard2%2 [04:31] heh, my system's still syncing [04:31] 72 GB in two days [04:32] oops, a bug [04:33] http://iabak.archiveteam.org:8080/render/?width=1060&height=731&_salt=1428467542.784&from=00%3A00_20150407&until=23%3A59_20150407&lineMode=connected&tz=UTC&logBase=&bgcolor=000000&majorGridLineColor=FFFFFF&fgcolor=FFFFFF&minorGridLineColor=C0C0C0&colorList=red%2Cgold%2Cgreen%2Cwhite&lineWidth=2&yMin=0&target=alias%28iabak.shardstats.numcopies.0.shard2%2C%220%20copies%22%29&target=alias%28sumSeries%28iabak.shardstats.numco [04:38] even better: [04:38] http://iabak.archiveteam.org:8080/render/?width=1060&height=731&_salt=1428467843.343&from=00%3A00_20150407&until=23%3A59_20150407&lineMode=staircase&tz=UTC&logBase=&bgcolor=000000&majorGridLineColor=FFFFFF&fgcolor=FFFFFF&minorGridLineColor=C0C0C0&colorList=red%2Cgold%2Cgreen%2Cwhite&lineWidth=2&yMin=0&target=legendValue%28alias%28keepLastValue%28iabak.shardstats.numcopies.0.shard2%29%2C%220%20copies%22%29%2C%22last%22%29& [04:38] make your own at http://iabak.archiveteam.org:8080 [04:42] time for me to sleep [05:18] *** Control-S has joined #internetarchive.bak [05:24] *** Ctrl-S has quit IRC (Read error: Operation timed out) [05:24] *** Control-S is now known as Ctrl-S [05:39] *** SN4T14_ has quit IRC (Ping timeout: 306 seconds) [05:45] *** SN4T14 has joined #internetarchive.bak [05:51] Just for the record, there's a mass of reasons for darking. Including: [05:51] - Rightholder request [05:51] *** zottelbey has joined #internetarchive.bak [05:51] - Item replaced by better item [05:51] - Mistake [05:51] - Temporarily stored away out of site until some problem is fixed [05:51] - Spam [05:52] - System file [06:41] *** niyaje4 has joined #internetarchive.bak [07:19] something went boom [07:53] *** niyaje4 has quit IRC (Ping timeout: 600 seconds) [08:40] *** ppiixx has joined #internetarchive.bak [09:27] *** Senji is now known as Senji2 [09:31] *** Senji has joined #internetarchive.bak [09:31] *** Senji2 has quit IRC (leaving) [12:05] *** atomotic has joined #internetarchive.bak [12:27] db48x: that solved my problem ;) thank you [12:27] hater: you're welcome. iten spotted it; too bad we haven't figured out why _his_ key isn't working :P [12:28] * hater is laughing [12:30] damn ssh ^ [12:30] might be similar to why your previous key didn't work; too bad we don't know why that didn't work [12:44] *** beardicus has joined #internetarchive.bak [12:48] *** Atluxity has joined #internetarchive.bak [13:00] db48x: install-git-annex gives an error if there is no internet connection [13:01] replacing line 8 with this solves the problem: if [ -n "$newVersion" ] && [ "$installedVersion" -lt "$newVersion" ] [13:02] i'm too lazy to update my fork, create a branch and then pull a request [14:18] the connected users file changed or isnt working anymore, thats why the map is empty btw [14:23] 'cor these fscks are ...slow... [14:35] Ok, so, let's quickly figure out why the graphs page went flat-a-roo [14:38] closure: The stats.tar.gz that is downloaded from iabak has zero-length geolists. [14:38] db48x: Unless you're generating the files now. [15:05] *** atomotic has quit IRC (Quit: Textual IRC Client: www.textualapp.com) [15:09] I touched the script that generates them [15:09] it works when I run it myself [15:10] let's see if I can figure it out now that I've slept and eaten and taken a pain killer [15:14] yep, I see the problem :) [15:14] You did it, you killed it [15:15] Blood on your hands [15:15] mixing with your already extent blood [15:15] yep [15:15] https://github.com/db48x/IA.BAK/commit/61774ca0a798c8f079a7ac05dcc3542da2451fb3#diff-98514952a623b3aeedab83f2e5c6d08eR85 [15:16] I saw this last night, but didn't think it was the cause [15:16] just noted it as something else I needed to fix [15:17] My fast fsck took 3h7m; the non-fast fsck is still running (about 1h30 in). I don't have a huge amount of shard1 though so I don't have any idea how that figure would scale if I had more [15:19] Senji: 3 hours isn't very fast! [15:20] Senji: how much do you have? [15:20] Indeed, and it looks as if it's IO bound, not CPU bound (the machine it's running on isn't very beefy; so CPU boundness could be a problem) [15:21] sep332: if I run du on shard1 now that's going to adversely affect the normal-fsck time :) [15:21] heh, probably not [15:22] either the data du wants is already in your disk cache, in which case it's free, or it's data that fsck will eventually want, in which case it's free [15:22] I think I have about a tenth of it though [15:22] git annex find --in=here will list all the files you have [15:30] db48x: that's only true if I have enough buffer cache available :-). I'm a bit disbelieving that things will stay in buffer cache for 3 hours :) [15:32] quite possibly true :) [15:35] Presumably I should avoid downloading any particular shard onto more than one machine; because otherwise I might end up with two copies of the same file (which, on machines less than a metre apart might be a bad idea) [15:39] yea, that's a risk [15:39] you can sync the file lists manually if you want, though [15:41] use git annex find --in=here on one, then use git annex drop on the other to drop that list of files [15:41] I've still got quite a bit of space to fill up on this machine; I'll just wait until shard3 for the other one :-) [15:41] hmm [15:42] in fact, I bet you could set up a rule to make each repository fail to want anything which specific other respositories already have [15:43] git annex wanted --not --in=otherrepo [15:44] db48x: how's your hand? [15:46] nothing's fallen off, and I can type if I've taken something for the pain recently [15:46] SketchCow: yay :) http://iabak.archiveteam.org/stats/ALL.geolist [15:48] I think you've got my "city" there from my ISP's name (which is "Andrews & Arnold") :-) [15:48] heh [15:48] blame freegeoip.net [15:52] ok, sounds like we should run fast fsck less frequently [15:52] although on an ssd, I can fsck in around 12 minutes I think [15:52] closure: I can't imagine my disks are that much slower; maybe it is CPU-bound and just not when I'm looking at top [15:53] seek-bound [16:03] * db48x sighs [16:04] I should tweak shardstats so that it doesn't have to actually call git annex info every time I want to test it [16:23] *** Start-mob has joined #internetarchive.bak [16:26] *** Start sets mode: +o Start-mob [16:40] *** Start has quit IRC (Disconnected.) [16:40] I have 10222 files in shard1 [16:55] *** Start-mob has quit IRC (Remote host closed the connection) [16:55] *** Start-mob has joined #internetarchive.bak [16:55] *** svchfoo1 sets mode: +o Start-mob [17:00] db48x: it's scary how close it is to where i really am [17:01] closure: fsck fast? i can test that on my dataset to see how fast it goes [17:02] *** Start-mob has quit IRC (Ping timeout: 370 seconds) [17:05] ah iabak runs that [17:06] midas: there's a reason we call them ICBM coordinates ;) [17:07] aye :p [17:07] freegeoip just sank my battleship [17:10] db48x: do all pubkeys for shard1 also have access to shard2? [17:14] not automatically, but I've been adding keys to both at the same time [17:37] So, here's my fast fsck vs fsck stats: http://pastebin.ubuntu.com/10774937/ [17:42] Looks like the graphs work again - thanks, db48x [17:46] SketchCow: you're welcome [17:50] we should be doing >&- 2>&- there, rather than /dev/null [17:51] shouldn't have a huge effect on the time though [17:52] oops, infinite loop :P [17:56] protip: always increment your loop variable [17:56] if it's worth looping, it's worth looping forever, right? [17:58] :) [17:58] one minute you want a loop, the next minute you don't... just make up your mind [18:04] The big question is if the "connected users" are meant to no longer be sawtooth" [18:07] SketchCow: hard to say if it was ever mean to be a sawtooth in the first place :) [18:11] maybe someone changed the munin configuration? I haven't [18:38] db48x: are you working on git-annex? [18:41] ask what you want to ask instead [18:41] running more iabak-instances at once results in a very high cpu-load [18:41] ask, don't ask to ask :-) [18:41] hater: well, of course [18:42] if the parallel feature in git-annex is coming soon, there will be no need to open a issue ticket [18:42] 'cos that's what cpu-load *measures* [18:42] fastfsck is (when running multiple instances) more like a bug than a feature [18:43] I think I convinced closure earlier that fast fsck wants to run less often :) [18:43] oh [18:44] *** Start has joined #internetarchive.bak [18:44] (and fastfsck throws an error after some time because there are other fsck-instances running(and locking)) [18:45] *** Start has quit IRC (Read error: Connection reset by peer) [18:45] *** Start has joined #internetarchive.bak [18:45] Also the hourly git updates tend to squash each other [18:45] I haven't really done any work on git annex itself [18:45] I've perused the source a little [18:45] joey however, although he's not in here [18:46] Senji: squash? [18:46] i think we should write down the issues or some of them will get lost during development [18:46] github tickets work well enough at this scale :) [18:47] db48x: if you start 10 copies of iabak now then in an hour it runs 10 sets of the git update; and they either run very slow or some of them fail [18:47] mmm, very good point [18:47] Senji: is simple .file-lock should work [18:48] it's not super critical that every sync succeed, but we should only sync each shard once per hour, no matter how often you start iabak [18:48] with-lock-ex is often your friend :) [18:55] *** Start has quit IRC (Disconnected.) [19:07] *** Start has joined #internetarchive.bak [19:23] the only reason to run iabak in parallel is to download faster, right? [19:23] so parellelizing git-annex get would do that [19:24] and you'd only need to run iabak once, maybe with a flag for how many downloads to do at once [19:25] *** Start has quit IRC (Disconnected.) [19:42] My next question is if shard2's growth/downloads are being accurately reflected. [19:42] http://iabackup.archiveteam.org/ia.bak/SHARD2 [19:42] That seems really, really slow [19:46] *** Start-mob has joined #internetarchive.bak [19:46] *** svchfoo1 sets mode: +o Start-mob [19:49] *** SN4T14_ has joined #internetarchive.bak [19:53] *** SN4T14 has quit IRC (Ping timeout: 306 seconds) [19:59] *** Start-mob has quit IRC (Remote host closed the connection) [20:01] *** atomotic has joined #internetarchive.bak [20:03] hi there. i could run iabak [20:04] who can give me access to SHARD2@iabak.archiveteam.org ? [20:05] Hang around and you'll get it [20:07] ok [20:09] *** Start-mob has joined #internetarchive.bak [20:09] *** svchfoo3 sets mode: +o Start-mob [20:17] Senji: hey, could you try this: time git annex fsck --fast --in here [20:18] my theory is that might be enough faster to use it [20:18] *** Start-mob has quit IRC (Leaving) [20:18] *** Start-mob has joined #internetarchive.bak [20:19] *** svchfoo1 sets mode: +o Start-mob [20:19] Trying that [20:21] If it takes as long as earlier I'll be asleep when it finishes though :) [20:21] here, it took 2 minutes [20:22] vs 12 before [20:22] will depend on the amout of files you have [20:22] closure: can you add atomotic? [20:25] atomotic: msg me your key [20:25] db48x: oh, you de-sawtoothed it? How? I didn't understand why it was doing that in the munin graph [20:29] SketchCow: SHARD2 stats match what I see if I run "git annex info ." in a clone of SHARD2 [20:29] you can try that at home :) [20:29] I think that some people were tearing through, but their disks are full now [20:30] slow and steady wins the race; can't lose with Comcast [20:30] ah, good, we caught back up on SHARD1 with those 60k files I had to redo [20:30] hah [20:31] I think I'm about 80 gigs into shard1 [20:31] at this point we'd have filled up pre-doubler Johnny Mnemonic [20:32] *** Start has joined #internetarchive.bak [20:32] keep free> 1TB [20:32] numfmt: invalid suffix in input ‘1TB’: ‘B’ [20:32] ./iabak-helper: 101: [: -lt: argument expected [20:33] try tb, I think? [20:33] *** Start-mob has quit IRC (Leaving) [20:34] *** Start-mob has joined #internetarchive.bak [20:34] *** svchfoo1 sets mode: +o Start-mob [20:35] hmm, this numfmt thing does not seem to work for me either [20:36] maybe just 1T [20:36] aha, it only accepts "1T" [20:36] *** Start-mob has quit IRC (Remote host closed the connection) [20:37] closure: So it SOUNDS like I should scare up a few more people. [20:37] btw it started and is running [20:37] I wish we knew how much "committed space" there is - i.e. how much space people have, and how much they're using. [20:37] SketchCow: still have some settling in on shard1, need to get fscking and expiry going [20:37] so say it's a 1gb shard, and three people with 500gb drives are helping. [20:37] well, we know how much space people are using (it's a bit expensive to query it though) [20:38] And we know, then, that there's 3x499gb of "unused" [20:38] I have about another TB on my current machine; and another TB on another machine. [20:38] Like, the All-shard is currently 6.23tb. Of our gang of 31/32, how much more space is there? [20:38] yeah, I hear you on unused. Hmm, could do something with metadata to record that in git [20:39] And maybe a third terabyte if I pull out some old disks :) [20:39] Well, right now, we know that 32 clients have Shard 1, and nine have Shard 2. [20:39] Maybe we need to ask people who are backing up Shard 1 to add in Shard 2 if they haven't. [20:40] And we need a check to say "hey, you're keeping a second copy of same shard, sorry, no" [20:44] is there a way to tell git-annex to avoid files that are already on another box? [20:44] If I have Box A and Box B right next to each other, can I tell them to get different files? [20:44] sep332: yes, you can. [20:44] Yes. [20:44] one way is to write down that uuid and say "git annex get --not --in $uuid" [20:45] another way is to set up a git remote from one box, pointing at the repo in the other, and then you can say "git annex get --not --in $remote" [20:45] the uuid one looks perfect, thanks [20:46] (you'll still want --not --copies 4 too) [20:46] you can also move files between repos, etc [20:48] btw you get the uuid from git config annex.uuid [20:49] for multiples, do I need --not --in $uuidA --not --in $uuidB [20:49] or is that too many --not's [20:53] as many as you like [20:54] *** atomotic has quit IRC (Quit: My Mac has gone to sleep. ZZZzzz…) [20:54] *** zottelbey has quit IRC (Remote host closed the connection) [20:56] *** atomotic has joined #internetarchive.bak [21:04] joey@beaver:~/lib/backup/IA.BAK/shard1>time ../git-annex.linux/git-annex fsck --in . --fast --quiet [21:04] Bad file size (67 B larger); moved to .git/annex/bad/MD5-s2038--c09a81cc07e2ab2d0592d7aed9feaacb [21:04] that's why it's good to run these fscks! [21:04] probably http resume failure.. [21:06] *** acridAxid has quit IRC (Quit: Quitting) [21:09] *** acridAxid has joined #internetarchive.bak [21:09] *** svchfoo3 sets mode: +o acridAxid [21:09] real 4m25.007s [21:10] that's on a spinning disk, so not very bad for the fast fsck [21:10] I think my machine must just bee too slow :) [21:11] It's still going on its fast fsck; 52mins in :) [21:11] what kind of disk bus and filesystem is it? [21:12] SATA / RAID-1 / ext3 (with dir_index) [21:12] I think they're 7200 rpm disks [21:12] hmm, mine is SATA, ext4 [21:13] not over nfs or something is it? [21:13] *** atomotic has quit IRC (Quit: Textual IRC Client: www.textualapp.com) [21:14] No, that'd be horrible :) [21:14] *** atomotic has joined #internetarchive.bak [21:14] She Who Must Be Obeyed calls me away from the computer. [21:21] *** Start has quit IRC (Read error: Connection reset by peer) [21:21] *** Start has joined #internetarchive.bak [21:23] *** Start has quit IRC (Read error: Connection reset by peer) [21:23] *** Start has joined #internetarchive.bak [21:25] -------------------------------------------------------------------- [21:25] If you're running a client for just Shard 1, consider doing one for [21:25] Shard 2. We don't want to expand the test client pool too much and [21:26] hey, you got it working once. [21:26] iabak will automatically switch over to shard2 when it's run already [21:26] -------------------------------------------------------------------- [21:26] *** Start-mob has joined #internetarchive.bak [21:26] you say that and yet the clients do not match [21:26] I see 32 on one, 9 on the other [21:26] not everyone is running the script repeatedly [21:26] Maybe they need to know about a command [21:26] That counts [21:26] *** svchfoo2 sets mode: +o Start-mob [21:26] That counts as someone going "OK, how do I do that" [21:26] And you go "run the thing" [21:27] I didn't say the tech support was hard! [21:27] *** niyaje4 has joined #internetarchive.bak [21:38] *** atomotic has quit IRC (Quit: Textual IRC Client: www.textualapp.com) [21:42] *** Start-mob has quit IRC (Ping timeout: 370 seconds) [21:46] *** Start has quit IRC (Read error: Connection reset by peer) [21:46] *** Start_ has joined #internetarchive.bak [21:51] *** Start_ is now known as Start [22:23] *** Start has quit IRC (Disconnected.) [22:27] *** Start-mob has joined #internetarchive.bak [22:30] *** niyaje4 has quit IRC (Read error: Operation timed out) [22:34] *** Start-mob has quit IRC (Remote host closed the connection) [22:47] With --in here: 80 minutes: so more than twice as fast [22:52] better than nothing, but .. [22:52] if it's that slow for others, let's find a different way [22:54] if you want to hack your script locally, you can make it fsck --fast a single file or small directory. This still prevents your repo getting expired due to inactivity [22:55] *** Start has joined #internetarchive.bak [22:56] it'll still do a full fsck monthly [22:56] *** svchfoo3 sets mode: +o Start [22:57] Once I've mostly filled up my space on this machine it shouldn't be a problem -- and the lock so it only fscks once at a time helps a lot with running multiple copies [22:58] wait.. is your machine running other iabak's concurrently? [22:58] Not while I was testing [22:59] ok, ok [22:59] I'm just getting it to run 10 in parallel now; 'cos bandwidth is a lot cheaper overnight :-) [23:00] I really should finish git annex get -jN [23:00] I have everything except one little peice, and the progress bar library still needs a lot of work [23:00] maybe by monday :) [23:05] aha [23:05] yea [23:06] do the timestamps in this file have to be in this format? [23:07] that was simply the first half-way reasonable thing I came up with [23:07] and I broke the hostname, doh [23:07] I'd like to do journalctl --unit=ssh.service --format=short-iso --utc [23:09] *** Start-mob has joined #internetarchive.bak [23:09] well, I can turn on persistent journaling [23:09] then it can just use cut instead of perl :) [23:11] oops, forgot about zgrep [23:12] *** wp494_ has joined #internetarchive.bak [23:15] *** wp494 has quit IRC (Ping timeout: 740 seconds) [23:35] starting a fast fsck of shard1 now... [23:36] oh, done already... was actually about 4 minutes. [23:41] so i don't show up on the map. i don't feel sufficienty appreciated. is that code still seekrit? [23:45] durr. nevermind... see the server branch.