[00:32] *** primus105 has quit IRC (Leaving.) [00:43] Kazzy: yes [00:58] +6TB storage bonus! [00:58] should last about a week :D [01:12] *** Beardface has quit IRC (Quit: leaving) [01:35] closure: is it safe to remove .git/annex/tmp as long as the program is not running? [01:36] sure [01:36] will i lose any data? [01:36] only any partially completed downloads [01:52] *** VADemon has quit IRC (Quit: left4dead) [01:54] i'm up to /dev/sdl [02:08] i don't think the hourly sync is scaling very well [02:28] *** mariusz2 is now known as mks [02:31] *** Beardface has joined #internetarchive.bak [02:36] *** beardicus has quit IRC (Sleep.) [02:51] 03registrar 05master 9d7d571 06other 10SHARD6/pubkeys registration of relstnl on SHARD6 [02:54] 03registrar 05master b79a26f 06other 10SHARD3/pubkeys registration of tomhiggins on SHARD3 [02:54] 03registrar 05master 6ba9022 06other 10SHARD6/pubkeys registration of tomhiggins on SHARD6 [02:56] 03registrar 05master 151a486 06other 10SHARD1/pubkeys registration of relstnl on SHARD1 [02:56] 03registrar 05master 6fb3686 06other 10SHARD2/pubkeys registration of relstnl on SHARD2 [02:56] 03registrar 05master 226caaf 06other 10SHARD3/pubkeys registration of relstnl on SHARD3 [03:14] 03registrar 05master 0bf49c8 06other 10SHARD5/pubkeys registration of relstnl on SHARD5 [03:14] 03registrar 05master 0bf49c8 06other 10SHARD5/pubkeys registration of relstnl on SHARD5 [03:26] So much love [03:32] *** chfoo has quit IRC (Quit: chfoo) [04:34] *** zottelbey has joined #internetarchive.bak [05:04] *** zottelbey has quit IRC (Remote host closed the connection) [05:25] 03registrar 05master 6966394 06other 10SHARD6/pubkeys registration of ryan on SHARD6 [05:58] *** zz_CyberJ is now known as CyberJaco [06:29] *** primus104 has joined #internetarchive.bak [06:59] *** Beardface has quit IRC (Remote host closed the connection) [06:59] *** hendi_ has joined #internetarchive.bak [07:10] *** CyberJaco is now known as zz_CyberJ [07:12] *** primus104 has quit IRC (Leaving.) [07:15] *** Beardface has joined #internetarchive.bak [07:26] *** atomotic has joined #internetarchive.bak [07:41] Beginning to ponder if something is wrong with the registration of my instance as it's not showing up on the website at all. Is there anything I can do to verify that it's working? 'annex sync' works. [07:46] which shard? [07:46] and do you know what your uuid is? [07:57] shard6, ca44a4e3-3ff6-4e5c-a66f-5f813c8b11bc according to .git/config [08:27] You're in http://iabak.archiveteam.org/stats/SHARD6.leaderboard-raw but you're 30th on the list so not in the "top 25" [08:30] Senji: ah, good. I've been poking around the wrong files then I think. Wasn't expecting to break top25, but couldn't spot it anywhere. Cheers [08:40] there seem to be a huge number of people downloading on shard6 now; maybe closure should spin up some new shards [09:02] *** primus104 has joined #internetarchive.bak [09:10] *** ohhdemgir has quit IRC (Read error: Operation timed out) [09:17] midas (unregistered) 1.51TB [09:17] im so not registered [09:18] awesome, almost passing 3M GET [09:22] Well maybe you'd better register; before you get garbage collected :-) [09:30] i think i was registrated before there was registration :p [09:31] If you think about that logicallly... :-p [09:31] I discovered yesterday that 2 of my shard3s weren't registered and the other was... [09:58] there is a shard7 but it isnt in the repolist yet :) [09:58] midas: register! ;p [09:59] (run change-email in the iabak dir) [10:05] *** atomotic has quit IRC (Quit: Textual IRC Client: www.textualapp.com) [11:00] shard7 is in the repolist; it's just 'reserve' [11:03] ah so it is [11:05] ppiixx: cant connect to that box atm :p [11:05] oh wait [11:05] i might [11:06] 03registrar 05master 1ff96c3 06other 10SHARD1/pubkeys registration of midas on SHARD1 [11:06] 03registrar 05master 36ae9cb 06other 10SHARD2/pubkeys registration of midas on SHARD2 [11:06] yay [11:07] now we wait [11:09] *** atomotic has joined #internetarchive.bak [11:21] *** primus104 has quit IRC (Leaving.) [11:24] *** atomotic has quit IRC (Quit: Textual IRC Client: www.textualapp.com) [11:26] *** beardicus has joined #internetarchive.bak [11:26] Senji: we could activate shard7 [11:28] yay [11:28] moved to place 4 :p [11:30] closure: I think with 30 people downloading on shard6 we're likely to get a lot of race conditions leading to over-backuping of files [11:30] particularly since shard6 has some pretty big directories [11:30] yeah, we probably have enough, looking at the slope [11:32] http://iabak.archiveteam.org/stats/SHARD6 [11:32] *** marvinw has quit IRC (Ping timeout: 252 seconds) [11:33] would adding logic to avoid having more than four copies be too cpu intensive? [11:34] I should implement http://git-annex.branchable.com/design/balanced_preferred_content/ [11:34] There is logic to avoid having more than four copies [11:34] But there's a race where multiple clients download the same directory at the same time [11:34] *** marvinw has joined #internetarchive.bak [11:36] actually, if someone wants a nice haskell learning exersise, the mathy bits of that (or a better mathy bit) would be doable without knowing eg, monads :) [11:38] hmm my iabak seems to have fallen back to singlethreaded wget again [11:38] despite running with J15 [11:39] pikhq: currently, git-annex is passed batches of files (up to a thousand or so), at the end of a batch, it'll single-thread [11:39] ah seems to be a older hourlysync process from before i added the flag ignore me [11:40] * ppiixx kills some random processes :) [11:40] oh or that [11:40] yeah about hourly sync... [11:40] closure: also I think it should finish off anything in .git/annex/tmp before doing the random ordering stuff [11:41] it asks me for password to github, and since it's run in a script it will never get one. so it just freezez waiting for it [11:41] it probably should be changed to cronjob to get rid of sleep cmd as well [11:45] eh, why would you need a password for github? [11:45] i don't [11:45] but hourly does git annex sync [11:45] for some reason it asks me for pass [11:45] well, that doesn't use github. [11:46] if it's asking for a password, your iabak ssh key is not set up properly [11:47] well iabak registers my downloads so something is working.. [11:47] and i've got bot id_rsa files [11:47] *both [11:48] and the same thing is going at my other machine [11:57] closure: should ditch the '(Also, you can safely run more than one of these at a time, to use more bandwidth!) [11:57] message now and point peopel at ANNEXGETOPTS [12:06] *** mariusz has joined #internetarchive.bak [12:38] *** atomotic has joined #internetarchive.bak [12:51] 03registrar 05master 0851783 06other 10SHARD6/pubkeys registration of midas on SHARD6 [13:16] 03registrar 05master 2ac0090 06other 10SHARD6/pubkeys registration of karlo on SHARD6 [13:16] 03registrar 05master c5b5afd 06other 10SHARD7/pubkeys registration of karlo on SHARD7 [13:45] *** protodev has quit IRC (Read error: Operation timed out) [13:46] *** dirt has quit IRC (Read error: Operation timed out) [13:46] *** dirt has joined #internetarchive.bak [13:47] *** ivan` has quit IRC (Read error: Operation timed out) [13:51] *** protodev has joined #internetarchive.bak [13:53] *** primus104 has joined #internetarchive.bak [14:04] *** Start has quit IRC (Disconnected.) [14:10] *** ivan`_ has joined #internetarchive.bak [14:37] *** Senji_ has joined #internetarchive.bak [14:39] *** atomotic has quit IRC (Quit: Textual IRC Client: www.textualapp.com) [14:40] *** pikhq has quit IRC (Read error: Connection reset by peer) [14:40] *** Senji has quit IRC (Ping timeout: 506 seconds) [14:41] *** Senji_ is now known as Senji [14:43] *** primus104 has quit IRC (Leaving.) [14:59] *** pikhq has joined #internetarchive.bak [14:59] *** svchfoo2 sets mode: +o pikhq [15:22] 03registrar 05master 3f92130 06other 10SHARD7/pubkeys registration of info on SHARD7 [16:16] *** Start has joined #internetarchive.bak [16:18] 03registrar 05master 217c4f6 06other 10SHARD6/pubkeys registration of hendi on SHARD6 [16:19] 03registrar 05master 7b6e17c 06other 10SHARD7/pubkeys registration of hendi on SHARD7 [16:45] *** Start has quit IRC (Disconnected.) [16:48] *** Start has joined #internetarchive.bak [17:20] *** primus104 has joined #internetarchive.bak [17:26] *** Start has quit IRC (Read error: Connection reset by peer) [17:26] *** Start has joined #internetarchive.bak [17:28] need more nominations for shard8-9 http://archiveteam.org/index.php?title=INTERNETARCHIVE.BAK/nominations [17:29] I assume we'll need them by this weekend at current rates [17:29] of course, a grab-bag like shard7 is also possible [17:34] I just asked Brewster. [17:34] so brewster is fully onboard now? [17:38] *** sankin has joined #internetarchive.bak [17:42] *** Start has quit IRC (Disconnected.) [17:44] *** zottelbey has joined #internetarchive.bak [17:47] I'd say he's chin-rubbingly intrigued at what that fucking maniac dreamed up THIS time [17:47] But JSMESS paid off so whatevs [17:47] Anyway, he came back with "Prelinger, ask Prelinger First" [17:48] I asked Prelinger. He said he would be delighted for us to back his stuff up. [17:48] *** hendi_ has quit IRC (Ping timeout: 512 seconds) [17:48] So, please add prelinger [17:48] Prioritize, even. Make that the very next thing. [17:48] great [17:48] I realize it may take several shards. (Or not) [17:48] Or be f'in huge [17:48] But it's a crown jewel. [17:49] not that big iirc, I had it in a shard before [17:49] what does it consist of [17:51] home movies, instructional films, other ephemeral films https://archive.org/details/prelinger [17:51] ahh [17:52] Let me know when the Shard is active, I'll link Rick to it. [17:52] tons of people mine it for stock footage [17:55] Yes, it's a prominent, PROMINENT part of dozens of documentaries every year. [17:55] can you also add gopher mirror?;) [17:56] mariusz: http://archiveteam.org/index.php?title=INTERNETARCHIVE.BAK/nominations [17:57] Gopher collection is good [17:57] Problem is grabbing collections versus individual items [17:57] But it might be good to get individual large items that are precious [17:57] ok ok, i just need the secret word to sign up .. [17:58] yahoosucks [18:00] thanks:) [18:01] gopher mirror is in archiveteam-fire which we should probably do anyway [18:01] I think fire is huge [18:01] and transient [18:02] 4 TB [18:11] Also, I dreamed up something else [18:11] If you are backing up to a cold-storage item (this is in the future) then we want more copies out there. Possibly six. [18:55] *** Start has joined #internetarchive.bak [18:55] *** Start has quit IRC (Client Quit) [18:56] *** Start has joined #internetarchive.bak [19:14] *** ivan`_ is now known as ivan` [19:14] there will be a day that hyves is downloaded again to my box. [19:14] that day will be epic. [19:26] *** Start has quit IRC (Disconnected.) [19:36] *** zottelbey has quit IRC (Ping timeout: 512 seconds) [20:05] *** mariusz has quit IRC (Ping timeout: 483 seconds) [20:36] *** sankin has quit IRC (Leaving.) [22:06] *** primus105 has joined #internetarchive.bak [22:10] *** primus104 has quit IRC (Read error: Operation timed out) [22:33] *** zz_CyberJ is now known as CyberJaco [22:46] *** Start has joined #internetarchive.bak [23:35] *** primus105 has quit IRC (Read error: Operation timed out) [23:37] *** primus104 has joined #internetarchive.bak [23:57] *** CyberJaco is now known as zz_CyberJ