[00:18] *** bohdans has quit IRC (Read error: Connection reset by peer) [00:19] *** bohdans has joined #internetarchive.bak [00:35] so because my git sync failed because it was out of memory i'm getting an error (rerunning to get exact message) but it says that repack failed and i have to remove gc.log to get git annex sync to run but i couldn't find that file [00:41] thelsdj: yes, we should try to figure out why your flock is failing [00:41] so that it doesn't try to do the same thing twice [00:47] i notice that the git annex sync is flock -s so that means multiple could run right? [00:48] i ran git annex sync twice in a row on shard4, the second time it got: [00:48] fatal: Out of memory, getdelim failed [00:48] which seems weird since i would assume it would be doing the same stuff, and I have 4G swap now [00:49] i should probably add --debug to figure out which exact part of process its getting that on [00:51] *** VADemon has joined #internetarchive.bak [01:18] may be a bug in the git on my arm system [02:08] 03registrar 05master 7b75935 06other 10SHARD19/pubkeys registration of deewiant+ia.bak on SHARD19 [04:36] bah: [04:36] git-annex: : hGetContents: invalid argument (invalid byte sequence) [04:36] failed [04:36] (recording state in git...) [04:37] git-annex: fromkey: 1 failed [05:17] 03registrar 05master cd9923b 06other 10SHARD10/pubkeys registration of fusl on SHARD10 [05:18] 03registrar 05master 779db1e 06other 10SHARD16/pubkeys registration of fusl on SHARD16 [05:19] 03registrar 05master cc1c889 06other 10SHARD12/pubkeys registration of fusl on SHARD12 [05:19] 03registrar 05master cc1c889 06other 10SHARD12/pubkeys registration of fusl on SHARD12 [05:23] 03registrar 05master 6f72ce8 06other 10SHARD15/pubkeys registration of fusl on SHARD15 [05:23] 03registrar 05master 6f72ce8 06other 10SHARD15/pubkeys registration of fusl on SHARD15 [05:23] 03registrar 05master 369d659 06other 10SHARD12/pubkeys registration of fusl on SHARD12 [05:23] 03registrar 05master 369d659 06other 10SHARD12/pubkeys registration of fusl on SHARD12 [05:26] 03registrar 05master 95c462e 06other 10SHARD17/pubkeys registration of fusl on SHARD17 [05:26] 03registrar 05master 407be91 06other 10SHARD16/pubkeys registration of fusl on SHARD16 [05:27] keep free: 100G [05:27] ./iabak-helper: line 304: 100*1024*1024: command not found [05:27] ./iabak-helper: line 304: [: 100000000000: unary operator expected [05:27] commit [05:27] weird [05:31] 03registrar 05master 1a52d24 06other 10SHARD17/pubkeys registration of fusl on SHARD17 [05:31] 03registrar 05master 1a52d24 06other 10SHARD17/pubkeys registration of fusl on SHARD17 [05:31] 03registrar 05master 679e53d 06other 10SHARD17/pubkeys registration of fusl on SHARD17 [05:31] 03registrar 05master 679e53d 06other 10SHARD17/pubkeys registration of fusl on SHARD17 [05:31] 03registrar 05master eca09d9 06other 10SHARD10/pubkeys 10SHARD16/pubkeys registration of fusl on SHARD10 [05:31] 03registrar 05master eca09d9 06other 10SHARD10/pubkeys 10SHARD16/pubkeys registration of fusl on SHARD10 [05:31] 03registrar 05master 42a657c 06other 10SHARD10/pubkeys registration of fusl on SHARD10 [05:31] 03registrar 05master 6757efb 06other 10SHARD4/pubkeys registration of fusl on SHARD4 [05:31] 03registrar 05master 6757efb 06other 10SHARD4/pubkeys registration of fusl on SHARD4 [05:31] 03registrar 05master 6757efb 06other 10SHARD4/pubkeys registration of fusl on SHARD4 [05:31] 03registrar 05master 6757efb 06other 10SHARD4/pubkeys registration of fusl on SHARD4 [05:31] 03registrar 05master 6757efb 06other 10SHARD4/pubkeys registration of fusl on SHARD4 [05:31] 03registrar 05master 14d16f2 06other 10SHARD4/pubkeys registration of fusl on SHARD4 [05:31] 03registrar 05master 14d16f2 06other 10SHARD4/pubkeys registration of fusl on SHARD4 [05:31] 03registrar 05master 7317d2a 06other 10SHARD12/pubkeys registration of fusl on SHARD12 [05:31] 03registrar 05master 7317d2a 06other 10SHARD12/pubkeys registration of fusl on SHARD12 [05:31] 03registrar 05master a26b873 06other 10SHARD10/pubkeys registration of fusl on SHARD10 [05:31] 03registrar 05master a26b873 06other 10SHARD10/pubkeys registration of fusl on SHARD10 [05:31] 03registrar 05master 8e4bf74 06other 10SHARD19/pubkeys registration of fusl on SHARD19 [05:31] 03registrar 05master e7c7ddd 06other 10SHARD19/pubkeys registration of fusl on SHARD19 [05:31] 03registrar 05master e7c7ddd 06other 10SHARD19/pubkeys registration of fusl on SHARD19 [05:31] 03registrar 05master e7c7ddd 06other 10SHARD19/pubkeys registration of fusl on SHARD19 [05:31] 03registrar 05master f483875 06other 10SHARD15/pubkeys registration of fusl on SHARD15 [05:32] 03registrar 05master 176e0e6 06other 10SHARD3/pubkeys registration of fusl on SHARD3 [05:32] 03registrar 05master 176e0e6 06other 10SHARD3/pubkeys registration of fusl on SHARD3 [05:32] Hey heyyyyyyyyyyyyyyyyyy [05:32] 03registrar 05master a6969e6 06other 10SHARD18/pubkeys registration of fusl on SHARD18 [05:32] 03registrar 05master a6969e6 06other 10SHARD18/pubkeys registration of fusl on SHARD18 [05:32] 03registrar 05master a6969e6 06other 10SHARD18/pubkeys registration of fusl on SHARD18 [05:32] 03registrar 05master a58877f 06other 10SHARD16/pubkeys registration of fusl on SHARD16 [05:32] How much space do you have there [05:32] I think that's a bug [05:32] around 200T [05:32] 03registrar 05master a58877f 06other 10SHARD16/pubkeys registration of fusl on SHARD16 [05:33] db48x: is multi-registration a bug or expected? I forgot [05:34] Well, 1. thanks for the space [05:34] 2. Wow that's a lot of space, how do you have so much space [05:35] i have a bunch of hetzner servers that i use for majestic12 which only uses cpu and memory and around 200g of disk space [05:36] all those servers come with 4x4tb disks each, so yea, not much use for these [05:36] 03registrar 05master c0fe92f 06other 10SHARD4/pubkeys registration of fusl on SHARD4 [05:37] 03registrar 05master c296c19 06other 10SHARD10/pubkeys registration of fusl on SHARD10 [05:37] 03registrar 05master 41023ec 06other 10SHARD18/pubkeys registration of fusl on SHARD18 [05:38] yipdw: probably a bug [05:38] the shell script output above is a bit odd [05:39] also annoying:  db48x  ~  projects  IA.BAK-server  shard3  echo "MD5-s3263532--0b4d070eff7baa8ef314ca330aecb71f 78rpm/06CharlesTrenetPigeonVole/03 Charles Trénet - En quittant une ville.mp3" | git-annex fromkey [05:39] fromkey stdin [05:39] git-annex: : hGetContents: invalid argument (invalid byte sequence) [05:39] failed [05:39] git-annex: fromkey: 1 failed [05:39] oh heh [05:40] fusl: that is an awesome amount of space [05:40] also, line 304 is totally a bug [05:40] a quarter of the disk space is used by the ceph cluster i'm running on top of these things [05:41] :) [05:41] so i thought, why not share the disk space with iabak instead of leaving almost all disks empty [05:41] this bug had to have been put here by my enemies [05:41] is one of your enemies named Bash Sin Tax [05:42] yes [05:44] db48x: huh [05:44] $ echo "MD5-s3263532--0b4d070eff7baa8ef314ca330aecb71f 78rpm/06CharlesTrenetPigeonVole/03 Charles Trénet - En quittant une ville.mp3" | git-annex fromkey [05:44] fromkey stdin ok [05:44] (recording state in git...) [05:44] :( [05:44] what's your locale setting? [05:44] LANG=en_US.UTF-8 [05:44] huh that's weird [05:45] I ran fromkey in a clean git-annex repo, not sure if that's related [05:46] no luck [05:48] *** GLaDOS has joined #internetarchive.bak [05:54] *** GLaDOS has quit IRC (Quit: Oh crap, I died.) [05:55] *** GLaDOS has joined #internetarchive.bak [06:48] *** Start has quit IRC (Quit: Disconnected.) [07:08] Hey fusl - looks like we need more shards for you :p [07:31] 03registrar 05master e8cfd97 06other 10SHARD11/pubkeys registration of fusl on SHARD11 [07:52] Are fus1's shards working? [07:55] (Looks like it, with 1.64tb already in?) [08:06] this must be a git bug on arm or something: get MD5-s9216--4b7e32f37152e23346ad1d4c18f73ff6 fatal: Out of memory, getdelim failed [08:06] i've got 4G swap available [08:09] https://git-annex.branchable.com/forum/git-annex_add_out_of_memory_error/ [08:11] closure: any idea why I might be getting this? [08:55] *** kurt has quit IRC (Read error: Operation timed out) [09:14] 03registrar 05master c0a41b2 06other 10SHARD10/pubkeys registration of fusl on SHARD10 [09:22] *** VADemon has quit IRC (Read error: Connection reset by peer) [09:56] *** sevs has joined #internetarchive.bak [10:18] *** bohdans has quit IRC (Read error: Connection reset by peer) [10:18] *** bohdans has joined #internetarchive.bak [11:08] 03registrar 05master 61d6463 06other 10SHARD19/pubkeys registration of fusl on SHARD19 [11:34] *** Xibalba has quit IRC (Quit: ZNC 1.7.x-git-692-c55caba - http://znc.in) [11:35] *** Xibalba has joined #internetarchive.bak [14:00] *** Start has joined #internetarchive.bak [14:07] *** Start has quit IRC (Read error: Operation timed out) [15:01] *** Start has joined #internetarchive.bak [15:22] *** Start has quit IRC (Read error: Operation timed out) [15:27] *** VADemon has joined #internetarchive.bak [15:55] So - not sure what the best way to proceed here is. Got ~650gb downloaded. Running the script just loops through each shard, and eventually tells me there's nothing more to download [16:14] *** Start has joined #internetarchive.bak [16:37] *** Start has quit IRC (Read error: Operation timed out) [17:05] I'd like you and the fellas to discuss it, see if you've been assigned all you can be assigned. [17:08] Definitely something we want to nail down. I think fusl is also adding some good testing there [17:24] *** Start has joined #internetarchive.bak [17:31] db48x: you around? [18:09] https://static.kurtmclester.com/s/497I6954.png [18:09] this, i like this [18:20] It's definitely neat [18:20] I hope we don't have people with extra space who aren't getting the stuff! [18:20] (We don't have that many with a mass of space, though) [18:21] I'm having download issues, but then using ARM processors is a bad idea [18:21] With archivebot being large files, checksumming is slow, and it won't download the next set of files until it checksums what it's got [18:22] It's downloading 2 files, checksumming them which takes a few hours and then downloading the next 2 [18:23] http://iabak.archiveteam.org/status.html [18:23] now I just need to think of the best way to have this find all the shards [19:12] Something up with shad13 on that page? [19:14] It doesn't exist [19:14] uhh [19:23] yeah, shard13 isn't a thing [19:23] I still need someone to show me how to make a shard [20:24] DO NOT SPEAK OF SHARD13 [20:24] https://s-media-cache-ak0.pinimg.com/originals/75/01/ad/7501ad69e7b7c9c9d7f4e39adbd02f74.jpg [20:26] *** bohdans has quit IRC (Quit: This computer has gone to sleep) [20:33] *** SketchCow changes topic to: http://iabak.archiveteam.org/ | #archiveteam | are you registered? cron job working? check the webpage.. | DO NOT ASK ABOUT SHARD 13 [20:35] *** Muad-Dib has quit IRC (Quit: ZNC - http://znc.in) [21:49] what about shard$((14-1))? [21:49] has it never existed? [21:49] because of its number? [21:59] Oh, who knows [21:59] fusl: thanks for the space donation and your patience [21:59] I want to make sure everyone's using the full space they're donating to this [21:59] before we break it even more wide [22:01] it doesn't exist because it's the shard I was going to create, but never got round to [22:01] I think we'll probably leave that one alone [22:26] *** kyan has joined #internetarchive.bak [23:39] *** sevs has quit IRC (Ping timeout: 268 seconds) [23:52] 03registrar 05master 6206bf6 06other 10SHARD18/pubkeys registration of deewiant+ia.bak on SHARD18