[00:06] not that i can think of [01:05] *** kyan has quit IRC (Remote host closed the connection) [01:07] *** kyan has joined #internetarchive.bak [02:23] closure: what does git-annex: unable to decommit memory: Invalid argument mean? [03:13] sep332_: ping? [03:51] *** VADemon has quit IRC (Quit: left4dead) [04:15] brilliant error message: [04:15] aws-0.14.1 depends on conduit-1.2.8 which failed to install. [04:15] conduit-1.2.8 failed during the building phase. The exception was: [04:15] ExitFailure 1 [04:17] oh, the real error message exists, it's just further up [05:33] *** Start has quit IRC (Quit: Disconnected.) [05:35] *** Start has joined #internetarchive.bak [06:01] *** sep332 has joined #internetarchive.bak [06:02] *** sep332_ has quit IRC (Read error: Operation timed out) [06:51] *** kyan has quit IRC (Quit: Leaving) [06:57] *** sevs has joined #internetarchive.bak [07:24] The overall progress looks good [08:02] Im getting an error which says ive filled all my disk space, yet ive got over 5TB left [08:20] *** db48x` has joined #internetarchive.bak [08:21] HCross: can you show me a log? [08:22] *** db48x has quit IRC (Ping timeout: 255 seconds) [08:23] here as well: http://pastebin.com/2HPXWjmv [08:23] http://paste.nerds.io/vipijefeji.vhdl [08:24] its also only downloaded 82GB of Shard12 [08:24] started iabak again, seems to continue [08:24] interesting [08:24] it should have gone to a different active shard [08:24] mine doesnt, it says that every time [08:24] its on ARM if that helps [08:25] that won't matter, but other things will [08:25] *** db48x` is now known as db48x [08:25] are you using gnu awk? [08:26] I'm going to need a log made with +x [08:26] mawk [08:27] err, -x I mean [08:27] doing that [08:27] or try debugging with https://gist.github.com/db48x/b079eaf83d33361d28c8115e8e5352da [08:28] download that, then run 'source test.sh' [08:28] you can then call the functions yourself and see what they print out [08:28] diskfree and annexreserved and bytesFromSize [08:29] http://paste.nerds.io/ocadododog.hs [08:32] bytesFromSize $(diskfree) [08:32] bytesFromSize $(annexreserved) [08:33] outofspace and stillhavespace return true or false rather than printing a value [08:33] so check $? after calling them [08:33] 5100000000000 for 1 [08:33] 100000000 for 2 [08:34] looks like those work fine [08:35] what does outofspace; echo $? print? [08:35] oh, it's outofspace $(annexreserved); echo $? [08:36] 1 [08:36] ok, that means you're not out of space [08:36] what does stillhavespace; echo $? print? [08:37] 1 [08:37] aha [08:37] it disagrees [08:37] and that's the one we actually use when we decide whether to start over on a new shard or not [08:38] run set -x [08:38] then run stillhavespace again [08:38] ah ok, way to tell it to carry on? [08:38] and show me what you get [08:38] http://paste.nerds.io/jamomuqugo.lua [08:41] oh, duh [08:41] weird thing is though, that I tested his [08:41] this [08:41] and it did go on to another shard... [08:44] want me to try a reinstall of the OS and see how that goes? [08:44] no :) [08:45] I accidentally returned the inverse of the right answer [08:45] look at line 64 of test.sh [08:45] want SSH access? [08:45] ah oops [08:45] try this definition instead: [08:45] stillhavespace () { [08:45] ! outofspace "$(annexreserved)" [08:45] } [08:49] http://paste.nerds.io/urusolanun.lua [08:50] and what is $? [08:50] 127 [08:53] hrm [08:54] are you sure that's not the result of some other command? [08:55] http://harrycross.me/701.png [08:57] yep, that 127 is just the command not found [08:58] http://paste.nerds.io/joguyimixi.lua [08:59] looks good [09:00] use that to replace the definition of stillhavespace in iabak-helper [09:06] well, I must sleep [09:07] filed #53 to remind myself [09:07] goodnight [09:15] 03registrar 05master 4014abb 06other 10SHARD10/pubkeys registration of hcross on SHARD10 [09:36] 03registrar 05master a1a70a7 06other 10SHARD14/pubkeys registration of hcross on SHARD14 [09:53] hmmm same issue [09:53] iabak@IABAK-VM:/mnt/IABAK/IA.BAK$ df -Ph . Filesystem Size Used Avail Use% Mounted on 192.168.11.98:/mnt/user/IABAK 7.3T 2.0T 5.4T 27% /mnt/IABAK [09:54] "Filled up available disk space, so stopping here!" [09:55] http://pastebin.com/z4s3mPTw [09:56] iabak@IABAK-VM:/mnt/IABAK/IA.BAK$ du -sh 82G . [09:58] 03registrar 05master 3742ba3 06other 10SHARD9/pubkeys registration of hcross on SHARD9 [10:06] *** asktoomuc has quit IRC (Ping timeout: 268 seconds) [10:17] 03registrar 05master 4b24a13 06other 10SHARD16/pubkeys registration of hcross on SHARD16 [10:23] *** wp494 has quit IRC (Read error: Operation timed out) [10:28] *** atomotic has joined #internetarchive.bak [10:33] *** wp494 has joined #internetarchive.bak [11:20] file 003 is shard 17 [11:25] another 8TB of content to backup is ready [11:28] oh joy, Filled up available disk space, so stopping here! [11:28] kurt, time to start creating shards then :p [11:30] currently sitting just a laptop, I'm in no state to do productive things [11:30] but yeah, I'll need someone to walk me through this stuff at some point v soon please [11:31] sure, im just knocking out ArchiveBot shards which is just a case of running 2 scripts on each shard [11:37] *** asktoomuc has joined #internetarchive.bak [11:39] right [11:39] HCross: what did you drop into iabak-helper to get rid of the out of space error? [11:40] change stillhavespace for http://paste.nerds.io/bepafasixu.hs [11:44] 03registrar 05master bb37edb 06other 10SHARD9/pubkeys registration of Kaz on SHARD9 [11:44] looks like we're away, cheers [11:44] so if I change that I should be fine? [11:44] takes a time to ramp up to full download speed now ive found [11:45] yea [11:45] I can certainly try [11:46] I've had really odd download speeds from IA [11:46] anywhere from 30mbps an item, to 1mbps [11:46] it all depends on what IA node the file is on [11:51] funny, my client is showing as: IA.BAK CLIENT: removed [11:51] on the webpage: http://iabak.archiveteam.org/client/2fefc6557adf0e058e2c899d454f6075c0d79ba3.html [11:53] 03registrar 05master 0b92645 06other 10SHARD3/pubkeys registration of roninfight on SHARD3 [11:57] heh [11:57] do one item -> finished with this shard [11:57] 03registrar 05master 39f8a62 06other 10SHARD15/pubkeys registration of Kaz on SHARD15 [12:07] *** atomotic has quit IRC (Quit: Textual IRC Client: www.textualapp.com) [12:17] weird. I'm now capped at 10mbps across like 50 connections [12:18] 03registrar 05master 04cbb99 06other 10SHARD4/pubkeys registration of roninfight on SHARD4 [12:22] huh?? Why did my client switch to shard 3? shard 12 is hardly done [13:00] *** octobyt3 has joined #internetarchive.bak [13:01] Hi, I don't know if this is normal, but I'm having this spam my console: http://pastebin.com/raw/X9gcfK1v [13:01] shard4 [13:03] It's also saying for other shards I've tried to download that it has "finished downloading the shard". There's only ever a few files pulled down [13:03] Still got like 3.5TB space I want it to use [13:04] It failed trying to pull all those MD5s down [13:13] *** octobyt3 has quit IRC (Ping timeout: 268 seconds) [13:18] right [13:18] I am ready to make shards [13:26] *** octobyt3 has joined #internetarchive.bak [13:46] 03registrar 05master b9c80f8 06other 10SHARD17/pubkeys registration of hcross on SHARD17 [13:57] octobyt3: re earlier questions: the spam is normal and can be ignored [13:58] not sure what's up with "finished downloading the shard", but the client right now is a bit screwed up, should be fixed soon [13:58] if you restart ./iabak does it continue working on the same shard or does it switch to another? [14:00] sevs: It terminates [14:00] Although other shards have the same issue. Tried to do a manual checkout of shard16 yesterday and it kept saying complete, when it clearly wasn't [14:06] 03registrar 05master 8a39c74 06other 10SHARD10/pubkeys registration of roninfight on SHARD10 [14:07] as stupid as it sounds try again tomorrow, the software seems somewhat broken and the guy working on that sleeps right now [14:08] thats the best advice i can give you, don't want to break something else [14:20] sevs: yeah np, thanks [14:22] 03registrar 05master 40565f5 06other 10SHARD15/pubkeys registration of roninfight on SHARD15 [14:26] *** VADemon has joined #internetarchive.bak [14:44] *** octobyt3 has quit IRC (Ping timeout: 268 seconds) [15:32] *** cmaldonad has joined #internetarchive.bak [15:32] *** cmaldonad has quit IRC (Client Quit) [15:51] *** atomotic has joined #internetarchive.bak [16:12] HCross about? [16:16] *** atomotic has quit IRC (Quit: Textual IRC Client: www.textualapp.com) [16:30] hmmm I don't think ANNEXGETOPTS is working for me. I can only ever see 1 concurrent download, despite having -J9 in the file [16:31] asktoomuc: have you restarted the script in the last 17 hours? [16:31] there's been a fix for that [16:32] does the script updates on its own when you start it? [16:32] yes [16:33] because yes, I restarted it this morning after it failed again with the "no space" message [16:33] I changed the helper one earlier with HCross recommendation for the space issue [16:34] hmmm, the filesystem datestamp on "iabak" is for yesterday [16:34] so I guess it hasn't picked up the changes [16:34] what about iabak-helper? [16:35] or, more to the point, can you tell me what line 194 of iabak-helper is? [16:36] 194 ${GIT} -c annex.alwayscommit=false annex get ${ANNEXOPTS} -- "${files[@]:${offset}:${chunksize}}" [16:36] *** kyan has joined #internetarchive.bak [16:36] okay, yeah that's not updated [16:38] I wonder if my previous changes to the file prevented the update [16:38] is that possible? [16:38] yeah, git will complain if you do stuff like that [16:38] in the IABAK directory, run git stash, then git pull [16:39] I need to stop jumping between clients [16:42] 194 ${GIT} -c annex.alwayscommit=false annex get ${ANNEXGETOPTS} -- "${files[@]:${offset}:${chunksize}}" [16:42] yep, that's what we want [16:42] was there just a typo in the file name? [16:43] that's it https://github.com/ArchiveTeam/IA.BAK/commit/ccf95c376c3305a1bfcc629939d05dab88652ed9 [16:43] hopefully that also fixed the space problem or I'll have to redo the changes HCross gave me [16:44] you'll have to redo them I think [16:45] I haven't caught up with all of the discussion from last night, so not sure what the ideal changes are [16:45] and/or if the fix in place is the most ideal moving forward [16:45] he suggested changing the stillhavespace function by: http://paste.nerds.io/bepafasixu.hs [16:46] not sure if that's ideal but at least the script stopped complaining about space for me [16:46] yeah, I'm runnig the same fix atm. I'll leave that for db48x to decide, he's aware of it anyway [16:56] ok thanks :) [17:12] in other news, we have far too many active shards [17:27] db48x: > "unable to decommit memory" new one to me. Seems to be a mismatch between kernel and libc, ie the libc is trying to use a feature your kernel does not have [17:28] the "ancient" git-annex build might work better on such a machine.. the normal autobuilds may well have updated to a libc needing linux 4.5. [17:56] "too many active shards"? Is that a bad thing? [17:57] everytime I restart the script, it targets a different shard. Is that expected? [18:03] asktoomuc: it's sort of a bad thing. it makes management harder, because things have to be manually set to a "don't make more people download this" state [18:04] hmmm ok. I assumed the script could handle that on its own [18:05] same as to not spreading itself accross 10 different shards when restarting. This could probably be improved but I'm not going to be much help here apart from pointing the issue [18:05] 03registrar 05master 6f002b3 06other 10SHARD14/pubkeys registration of meroje on SHARD14 [18:44] 03registrar 05master f4391f2 06other 10SHARD11/pubkeys registration of hcross on SHARD11 [19:02] 03registrar 05master 30a3481 06other 10SHARD15/pubkeys registration of hcross on SHARD15 [20:09] db48x: the batching stuff I'm not sure is working properly, it goes for a few hours then stops, i've got like 4-6MB/s but isn't the point that it check in and keep going not just stop? [20:37] *** bwn has quit IRC (Ping timeout: 961 seconds) [20:43] *** Deewiant has joined #internetarchive.bak [20:48] *** bwn has joined #internetarchive.bak [21:15] *** asktoomuc has quit IRC (Quit: Page closed) [21:56] bah [21:57] my computer went to sleep and paused the test I was running [21:59] I'll just assume that it works and commit it [22:00] (after all, it's simple enough that it couldn't possibly be wrong) [22:07] famous last words [22:51] 03registrar 05master 549dde4 06other 10SHARD3/pubkeys registration of hcross on SHARD3 [22:58] So, I've been downloading shard9 [22:58] shard9 is now in maint [22:59] does iabak stop downloading more from that shard, should I tell it to? [23:09] 03registrar 05master cec9300 06other 10SHARD10/pubkeys registration of meroje on SHARD10 [23:09] it should do automatically [23:10] I switched it to maint because the only things it said didn't have 4 copies were the meta and torrent files - not sure why that isn't reflected on the graphs thougb [23:13] ok, thanks [23:14] I've now got a very sad shard9, not sure what's going on with it [23:27] Some of the shards have files in them that have been deleted from the IA and can't be backed up any more. I don't know if 9 is one of those. [23:28] db48x: any idea why after a few hours it just stops? I have to re-run iabak a couple times a day to get it to keep going, i assume it has to do with your chunking and checking in not working as intended [23:30] I probably can't help; but do you get an error message or anything else odd before it stops? [23:35] it stops saying that its finished getting the entire shard but it hasn't, if i re-run it gets more stuff [23:35] It might do that if it couldn't download the files for some reason. [23:37] 03registrar 05master 41c24be 06other 10SHARD4/pubkeys registration of hcross on SHARD4 [23:37] i think it has a time limit and stops after the time limit, but if i'm trying to get the whole shard i think it needs some logic to keep going [23:54] it's supposed to stop and move on to another shard, but there was a bug that made it think you were out of disk space, so it stoped completely [23:55] Kaz: how is your shard9 sad? [23:56] you can always go in to a shard and manually run ../git-annex.linux/git-annex get --your-critera-here and get whatever you fancy, right? [23:56] spent 30m or so doing a fsck of shard9, got to the end and realised it couldn't grab the meta files and whatnot, then just stalled and did nothing [23:56] that happened each time I ran the script [23:57] if you really want to have a complete personal backup of the shard, just say "git annex get" and it'll get everything it doesn't yet have [23:57] Kaz: hrm. it should never just do nothing [23:57] Kaz: can you add set -x to iabak-helper and then rerun it? [23:57] at least then we'll be able to see where it's stalling [23:57] brb, cake [23:57] it occured to me a few minutes ago though, that I had local changes in iabak-helper [23:58] but shard9 is active again now, and I've got to sleep at some point [23:58] db48x: seems like there should be a way for it to keep filling up a shard until i'm out of space? why would we not want it to do this? [23:58] I'll set it to maint again tomorrow and try again [23:58] as it is right now i can't use just './iabak' to do anything but have it download my existing shard for 8 hours then quit [23:58] because it won't start a new shard either