#internetarchive.bak 2016-11-20,Sun

↑back Search

Time Nickname Message
00:06 🔗 thelsdj not that i can think of
01:05 🔗 kyan has quit IRC (Remote host closed the connection)
01:07 🔗 kyan has joined #internetarchive.bak
02:23 🔗 db48x closure: what does git-annex: unable to decommit memory: Invalid argument mean?
03:13 🔗 db48x sep332_: ping?
03:51 🔗 VADemon has quit IRC (Quit: left4dead)
04:15 🔗 db48x brilliant error message:
04:15 🔗 db48x aws-0.14.1 depends on conduit-1.2.8 which failed to install.
04:15 🔗 db48x conduit-1.2.8 failed during the building phase. The exception was:
04:15 🔗 db48x ExitFailure 1
04:17 🔗 db48x oh, the real error message exists, it's just further up
05:33 🔗 Start has quit IRC (Quit: Disconnected.)
05:35 🔗 Start has joined #internetarchive.bak
06:01 🔗 sep332 has joined #internetarchive.bak
06:02 🔗 sep332_ has quit IRC (Read error: Operation timed out)
06:51 🔗 kyan has quit IRC (Quit: Leaving)
06:57 🔗 sevs has joined #internetarchive.bak
07:24 🔗 SketchCow The overall progress looks good
08:02 🔗 HCross Im getting an error which says ive filled all my disk space, yet ive got over 5TB left
08:20 🔗 db48x` has joined #internetarchive.bak
08:21 🔗 db48x` HCross: can you show me a log?
08:22 🔗 db48x has quit IRC (Ping timeout: 255 seconds)
08:23 🔗 sevs here as well: http://pastebin.com/2HPXWjmv
08:23 🔗 HCross http://paste.nerds.io/vipijefeji.vhdl
08:24 🔗 HCross its also only downloaded 82GB of Shard12
08:24 🔗 sevs started iabak again, seems to continue
08:24 🔗 db48x` interesting
08:24 🔗 db48x` it should have gone to a different active shard
08:24 🔗 HCross mine doesnt, it says that every time
08:24 🔗 HCross its on ARM if that helps
08:25 🔗 db48x` that won't matter, but other things will
08:25 🔗 db48x` is now known as db48x
08:25 🔗 db48x are you using gnu awk?
08:26 🔗 db48x I'm going to need a log made with +x
08:26 🔗 HCross mawk
08:27 🔗 db48x err, -x I mean
08:27 🔗 HCross doing that
08:27 🔗 db48x or try debugging with https://gist.github.com/db48x/b079eaf83d33361d28c8115e8e5352da
08:28 🔗 db48x download that, then run 'source test.sh'
08:28 🔗 db48x you can then call the functions yourself and see what they print out
08:28 🔗 db48x diskfree and annexreserved and bytesFromSize
08:29 🔗 HCross http://paste.nerds.io/ocadododog.hs
08:32 🔗 db48x bytesFromSize $(diskfree)
08:32 🔗 db48x bytesFromSize $(annexreserved)
08:33 🔗 db48x outofspace and stillhavespace return true or false rather than printing a value
08:33 🔗 db48x so check $? after calling them
08:33 🔗 HCross 5100000000000 for 1
08:33 🔗 HCross 100000000 for 2
08:34 🔗 db48x looks like those work fine
08:35 🔗 db48x what does outofspace; echo $? print?
08:35 🔗 db48x oh, it's outofspace $(annexreserved); echo $?
08:36 🔗 HCross 1
08:36 🔗 db48x ok, that means you're not out of space
08:36 🔗 db48x what does stillhavespace; echo $? print?
08:37 🔗 HCross 1
08:37 🔗 db48x aha
08:37 🔗 db48x it disagrees
08:37 🔗 db48x and that's the one we actually use when we decide whether to start over on a new shard or not
08:38 🔗 db48x run set -x
08:38 🔗 db48x then run stillhavespace again
08:38 🔗 HCross ah ok, way to tell it to carry on?
08:38 🔗 db48x and show me what you get
08:38 🔗 HCross http://paste.nerds.io/jamomuqugo.lua
08:41 🔗 db48x oh, duh
08:41 🔗 db48x weird thing is though, that I tested his
08:41 🔗 db48x this
08:41 🔗 db48x and it did go on to another shard...
08:44 🔗 HCross want me to try a reinstall of the OS and see how that goes?
08:44 🔗 db48x no :)
08:45 🔗 db48x I accidentally returned the inverse of the right answer
08:45 🔗 db48x look at line 64 of test.sh
08:45 🔗 HCross want SSH access?
08:45 🔗 HCross ah oops
08:45 🔗 db48x try this definition instead:
08:45 🔗 db48x stillhavespace () {
08:45 🔗 db48x ! outofspace "$(annexreserved)"
08:45 🔗 db48x }
08:49 🔗 HCross http://paste.nerds.io/urusolanun.lua
08:50 🔗 db48x and what is $?
08:50 🔗 HCross 127
08:53 🔗 db48x hrm
08:54 🔗 db48x are you sure that's not the result of some other command?
08:55 🔗 HCross http://harrycross.me/701.png
08:57 🔗 db48x yep, that 127 is just the command not found
08:58 🔗 HCross http://paste.nerds.io/joguyimixi.lua
08:59 🔗 db48x looks good
09:00 🔗 db48x use that to replace the definition of stillhavespace in iabak-helper
09:06 🔗 db48x well, I must sleep
09:07 🔗 db48x filed #53 to remind myself
09:07 🔗 HCross goodnight
09:15 🔗 iabak-reg 03registrar 05master 4014abb 06other 10SHARD10/pubkeys registration of hcross on SHARD10
09:36 🔗 iabak-reg 03registrar 05master a1a70a7 06other 10SHARD14/pubkeys registration of hcross on SHARD14
09:53 🔗 asktoomuc hmmm same issue
09:53 🔗 asktoomuc iabak@IABAK-VM:/mnt/IABAK/IA.BAK$ df -Ph . Filesystem Size Used Avail Use% Mounted on 192.168.11.98:/mnt/user/IABAK 7.3T 2.0T 5.4T 27% /mnt/IABAK
09:54 🔗 asktoomuc "Filled up available disk space, so stopping here!"
09:55 🔗 asktoomuc http://pastebin.com/z4s3mPTw
09:56 🔗 asktoomuc iabak@IABAK-VM:/mnt/IABAK/IA.BAK$ du -sh 82G .
09:58 🔗 iabak-reg 03registrar 05master 3742ba3 06other 10SHARD9/pubkeys registration of hcross on SHARD9
10:06 🔗 asktoomuc has quit IRC (Ping timeout: 268 seconds)
10:17 🔗 iabak-reg 03registrar 05master 4b24a13 06other 10SHARD16/pubkeys registration of hcross on SHARD16
10:23 🔗 wp494 has quit IRC (Read error: Operation timed out)
10:28 🔗 atomotic has joined #internetarchive.bak
10:33 🔗 wp494 has joined #internetarchive.bak
11:20 🔗 HCross file 003 is shard 17
11:25 🔗 HCross another 8TB of content to backup is ready
11:28 🔗 kurt oh joy, Filled up available disk space, so stopping here!
11:28 🔗 HCross kurt, time to start creating shards then :p
11:30 🔗 kurt currently sitting just a laptop, I'm in no state to do productive things
11:30 🔗 kurt but yeah, I'll need someone to walk me through this stuff at some point v soon please
11:31 🔗 HCross sure, im just knocking out ArchiveBot shards which is just a case of running 2 scripts on each shard
11:37 🔗 asktoomuc has joined #internetarchive.bak
11:39 🔗 kurt right
11:39 🔗 kurt HCross: what did you drop into iabak-helper to get rid of the out of space error?
11:40 🔗 HCross change stillhavespace for http://paste.nerds.io/bepafasixu.hs
11:44 🔗 iabak-reg 03registrar 05master bb37edb 06other 10SHARD9/pubkeys registration of Kaz on SHARD9
11:44 🔗 kurt looks like we're away, cheers
11:44 🔗 asktoomuc so if I change that I should be fine?
11:44 🔗 HCross takes a time to ramp up to full download speed now ive found
11:45 🔗 HCross yea
11:45 🔗 asktoomuc I can certainly try
11:46 🔗 kurt I've had really odd download speeds from IA
11:46 🔗 kurt anywhere from 30mbps an item, to 1mbps
11:46 🔗 HCross it all depends on what IA node the file is on
11:51 🔗 asktoomuc funny, my client is showing as: IA.BAK CLIENT: removed
11:51 🔗 asktoomuc on the webpage: http://iabak.archiveteam.org/client/2fefc6557adf0e058e2c899d454f6075c0d79ba3.html
11:53 🔗 iabak-reg 03registrar 05master 0b92645 06other 10SHARD3/pubkeys registration of roninfight on SHARD3
11:57 🔗 kurt heh
11:57 🔗 kurt do one item -> finished with this shard
11:57 🔗 iabak-reg 03registrar 05master 39f8a62 06other 10SHARD15/pubkeys registration of Kaz on SHARD15
12:07 🔗 atomotic has quit IRC (Quit: Textual IRC Client: www.textualapp.com)
12:17 🔗 kurt weird. I'm now capped at 10mbps across like 50 connections
12:18 🔗 iabak-reg 03registrar 05master 04cbb99 06other 10SHARD4/pubkeys registration of roninfight on SHARD4
12:22 🔗 asktoomuc huh?? Why did my client switch to shard 3? shard 12 is hardly done
13:00 🔗 octobyt3 has joined #internetarchive.bak
13:01 🔗 octobyt3 Hi, I don't know if this is normal, but I'm having this spam my console: http://pastebin.com/raw/X9gcfK1v
13:01 🔗 octobyt3 shard4
13:03 🔗 octobyt3 It's also saying for other shards I've tried to download that it has "finished downloading the shard". There's only ever a few files pulled down
13:03 🔗 octobyt3 Still got like 3.5TB space I want it to use
13:04 🔗 octobyt3 It failed trying to pull all those MD5s down
13:13 🔗 octobyt3 has quit IRC (Ping timeout: 268 seconds)
13:18 🔗 Kaz right
13:18 🔗 Kaz I am ready to make shards
13:26 🔗 octobyt3 has joined #internetarchive.bak
13:46 🔗 iabak-reg 03registrar 05master b9c80f8 06other 10SHARD17/pubkeys registration of hcross on SHARD17
13:57 🔗 sevs octobyt3: re earlier questions: the spam is normal and can be ignored
13:58 🔗 sevs not sure what's up with "finished downloading the shard", but the client right now is a bit screwed up, should be fixed soon
13:58 🔗 sevs if you restart ./iabak does it continue working on the same shard or does it switch to another?
14:00 🔗 octobyt3 sevs: It terminates
14:00 🔗 octobyt3 Although other shards have the same issue. Tried to do a manual checkout of shard16 yesterday and it kept saying complete, when it clearly wasn't
14:06 🔗 iabak-reg 03registrar 05master 8a39c74 06other 10SHARD10/pubkeys registration of roninfight on SHARD10
14:07 🔗 sevs as stupid as it sounds try again tomorrow, the software seems somewhat broken and the guy working on that sleeps right now
14:08 🔗 sevs thats the best advice i can give you, don't want to break something else
14:20 🔗 octobyt3 sevs: yeah np, thanks
14:22 🔗 iabak-reg 03registrar 05master 40565f5 06other 10SHARD15/pubkeys registration of roninfight on SHARD15
14:26 🔗 VADemon has joined #internetarchive.bak
14:44 🔗 octobyt3 has quit IRC (Ping timeout: 268 seconds)
15:32 🔗 cmaldonad has joined #internetarchive.bak
15:32 🔗 cmaldonad has quit IRC (Client Quit)
15:51 🔗 atomotic has joined #internetarchive.bak
16:12 🔗 Kaz HCross about?
16:16 🔗 atomotic has quit IRC (Quit: Textual IRC Client: www.textualapp.com)
16:30 🔗 asktoomuc hmmm I don't think ANNEXGETOPTS is working for me. I can only ever see 1 concurrent download, despite having -J9 in the file
16:31 🔗 kurt asktoomuc: have you restarted the script in the last 17 hours?
16:31 🔗 kurt there's been a fix for that
16:32 🔗 asktoomuc does the script updates on its own when you start it?
16:32 🔗 Kaz yes
16:33 🔗 asktoomuc because yes, I restarted it this morning after it failed again with the "no space" message
16:33 🔗 asktoomuc I changed the helper one earlier with HCross recommendation for the space issue
16:34 🔗 asktoomuc hmmm, the filesystem datestamp on "iabak" is for yesterday
16:34 🔗 asktoomuc so I guess it hasn't picked up the changes
16:34 🔗 Kaz what about iabak-helper?
16:35 🔗 Kaz or, more to the point, can you tell me what line 194 of iabak-helper is?
16:36 🔗 asktoomuc 194 ${GIT} -c annex.alwayscommit=false annex get ${ANNEXOPTS} -- "${files[@]:${offset}:${chunksize}}"
16:36 🔗 kyan has joined #internetarchive.bak
16:36 🔗 Kaz okay, yeah that's not updated
16:38 🔗 asktoomuc I wonder if my previous changes to the file prevented the update
16:38 🔗 asktoomuc is that possible?
16:38 🔗 kurt yeah, git will complain if you do stuff like that
16:38 🔗 kurt in the IABAK directory, run git stash, then git pull
16:39 🔗 Kaz I need to stop jumping between clients
16:42 🔗 asktoomuc 194 ${GIT} -c annex.alwayscommit=false annex get ${ANNEXGETOPTS} -- "${files[@]:${offset}:${chunksize}}"
16:42 🔗 Kaz yep, that's what we want
16:42 🔗 asktoomuc was there just a typo in the file name?
16:43 🔗 yipdw that's it https://github.com/ArchiveTeam/IA.BAK/commit/ccf95c376c3305a1bfcc629939d05dab88652ed9
16:43 🔗 asktoomuc hopefully that also fixed the space problem or I'll have to redo the changes HCross gave me
16:44 🔗 Kaz you'll have to redo them I think
16:45 🔗 Kaz I haven't caught up with all of the discussion from last night, so not sure what the ideal changes are
16:45 🔗 Kaz and/or if the fix in place is the most ideal moving forward
16:45 🔗 asktoomuc he suggested changing the stillhavespace function by: http://paste.nerds.io/bepafasixu.hs
16:46 🔗 asktoomuc not sure if that's ideal but at least the script stopped complaining about space for me
16:46 🔗 Kaz yeah, I'm runnig the same fix atm. I'll leave that for db48x to decide, he's aware of it anyway
16:56 🔗 asktoomuc ok thanks :)
17:12 🔗 Kaz in other news, we have far too many active shards
17:27 🔗 closure db48x: > "unable to decommit memory" new one to me. Seems to be a mismatch between kernel and libc, ie the libc is trying to use a feature your kernel does not have
17:28 🔗 closure the "ancient" git-annex build might work better on such a machine.. the normal autobuilds may well have updated to a libc needing linux 4.5.
17:56 🔗 asktoomuc "too many active shards"? Is that a bad thing?
17:57 🔗 asktoomuc everytime I restart the script, it targets a different shard. Is that expected?
18:03 🔗 Kaz asktoomuc: it's sort of a bad thing. it makes management harder, because things have to be manually set to a "don't make more people download this" state
18:04 🔗 asktoomuc hmmm ok. I assumed the script could handle that on its own
18:05 🔗 asktoomuc same as to not spreading itself accross 10 different shards when restarting. This could probably be improved but I'm not going to be much help here apart from pointing the issue
18:05 🔗 iabak-reg 03registrar 05master 6f002b3 06other 10SHARD14/pubkeys registration of meroje on SHARD14
18:44 🔗 iabak-reg 03registrar 05master f4391f2 06other 10SHARD11/pubkeys registration of hcross on SHARD11
19:02 🔗 iabak-reg 03registrar 05master 30a3481 06other 10SHARD15/pubkeys registration of hcross on SHARD15
20:09 🔗 thelsdj db48x: the batching stuff I'm not sure is working properly, it goes for a few hours then stops, i've got like 4-6MB/s but isn't the point that it check in and keep going not just stop?
20:37 🔗 bwn has quit IRC (Ping timeout: 961 seconds)
20:43 🔗 Deewiant has joined #internetarchive.bak
20:48 🔗 bwn has joined #internetarchive.bak
21:15 🔗 asktoomuc has quit IRC (Quit: Page closed)
21:56 🔗 db48x bah
21:57 🔗 db48x my computer went to sleep and paused the test I was running
21:59 🔗 db48x I'll just assume that it works and commit it
22:00 🔗 db48x (after all, it's simple enough that it couldn't possibly be wrong)
22:07 🔗 Xibalba famous last words
22:51 🔗 iabak-reg 03registrar 05master 549dde4 06other 10SHARD3/pubkeys registration of hcross on SHARD3
22:58 🔗 sevs So, I've been downloading shard9
22:58 🔗 sevs shard9 is now in maint
22:59 🔗 sevs does iabak stop downloading more from that shard, should I tell it to?
23:09 🔗 iabak-reg 03registrar 05master cec9300 06other 10SHARD10/pubkeys registration of meroje on SHARD10
23:09 🔗 Kaz it should do automatically
23:10 🔗 Kaz I switched it to maint because the only things it said didn't have 4 copies were the meta and torrent files - not sure why that isn't reflected on the graphs thougb
23:13 🔗 sevs ok, thanks
23:14 🔗 Kaz I've now got a very sad shard9, not sure what's going on with it
23:27 🔗 Senji Some of the shards have files in them that have been deleted from the IA and can't be backed up any more. I don't know if 9 is one of those.
23:28 🔗 thelsdj db48x: any idea why after a few hours it just stops? I have to re-run iabak a couple times a day to get it to keep going, i assume it has to do with your chunking and checking in not working as intended
23:30 🔗 Senji I probably can't help; but do you get an error message or anything else odd before it stops?
23:35 🔗 thelsdj it stops saying that its finished getting the entire shard but it hasn't, if i re-run it gets more stuff
23:35 🔗 Senji It might do that if it couldn't download the files for some reason.
23:37 🔗 iabak-reg 03registrar 05master 41c24be 06other 10SHARD4/pubkeys registration of hcross on SHARD4
23:37 🔗 thelsdj i think it has a time limit and stops after the time limit, but if i'm trying to get the whole shard i think it needs some logic to keep going
23:54 🔗 db48x it's supposed to stop and move on to another shard, but there was a bug that made it think you were out of disk space, so it stoped completely
23:55 🔗 db48x Kaz: how is your shard9 sad?
23:56 🔗 db48x you can always go in to a shard and manually run ../git-annex.linux/git-annex get --your-critera-here and get whatever you fancy, right?
23:56 🔗 Kaz spent 30m or so doing a fsck of shard9, got to the end and realised it couldn't grab the meta files and whatnot, then just stalled and did nothing
23:56 🔗 Kaz that happened each time I ran the script
23:57 🔗 db48x if you really want to have a complete personal backup of the shard, just say "git annex get" and it'll get everything it doesn't yet have
23:57 🔗 db48x Kaz: hrm. it should never just do nothing
23:57 🔗 db48x Kaz: can you add set -x to iabak-helper and then rerun it?
23:57 🔗 db48x at least then we'll be able to see where it's stalling
23:57 🔗 db48x brb, cake
23:57 🔗 Kaz it occured to me a few minutes ago though, that I had local changes in iabak-helper
23:58 🔗 Kaz but shard9 is active again now, and I've got to sleep at some point
23:58 🔗 thelsdj db48x: seems like there should be a way for it to keep filling up a shard until i'm out of space? why would we not want it to do this?
23:58 🔗 Kaz I'll set it to maint again tomorrow and try again
23:58 🔗 thelsdj as it is right now i can't use just './iabak' to do anything but have it download my existing shard for 8 hours then quit
23:58 🔗 thelsdj because it won't start a new shard either
irclogger-viewer