[00:13] *** protodev has quit IRC (Ping timeout: 606 seconds)
[00:14] *** protodev has joined #internetarchive.bak
[00:32] *** primus104 has quit IRC (Leaving.)
[00:40] *** protodev has quit IRC (Ping timeout: 606 seconds)
[00:41] *** protodev has joined #internetarchive.bak
[00:56] *** niyaje4 has joined #internetarchive.bak
[01:10] *** protodev has quit IRC (Ping timeout: 606 seconds)
[01:10] *** protodev has joined #internetarchive.bak
[01:21] *** protodev has quit IRC (Ping timeout: 606 seconds)
[01:22] *** protodev has joined #internetarchive.bak
[04:21] *** niyaje4 has quit IRC (Ping timeout: 600 seconds)
[05:03] *** Ctrl-S has quit IRC ( HydraIRC -> http://www.hydrairc.com <- In tests, 0x09 out of 0x0A l33t h4x0rz prefer it :))
[05:10] *** Ctrl-S has joined #internetarchive.bak
[05:48] *** marvinw has quit IRC (Read error: Operation timed out)
[06:08] *** zottelbey has joined #internetarchive.bak
[06:13] *** marvinw has joined #internetarchive.bak
[06:15] *** Control-S has joined #internetarchive.bak
[06:19] *** Ctrl-S has quit IRC (Read error: Operation timed out)
[06:19] *** Control-S is now known as Ctrl-S
[06:20] *** primus104 has joined #internetarchive.bak
[06:22] *** niyaje4 has joined #internetarchive.bak
[07:19] *** atomotic has joined #internetarchive.bak
[07:36] <lhobas> my SHARD3 download finished, with 108 files that could not be downloaded - any similar experiences here?
[08:15] <Senji> There's a bug with URL parsing of files with # in the URLs
[08:23] *** niyaje4 has quit IRC (Ping timeout: 600 seconds)
[09:07] *** atomotic has quit IRC (Quit: Textual IRC Client: www.textualapp.com)
[09:28] <Senji> So, when's shard5 going to appear? :-)
[10:50] *** primus104 has quit IRC (Leaving.)
[11:00] <db48x> neat, one of my shards became corrupted
[11:00] <db48x> https://pastebin.mozilla.org/8832598
[11:00] <Senji> neat?
[11:00] <db48x> git annex fsck fixed it though
[11:01] <db48x> Senji: yes
[11:01] <db48x> this is a test, we have to have bad things happen to it
[11:03] <Senji> Good point :)
[11:03] <db48x> I deliberately put it on a disk that I suspected of having trouble
[11:04] <Senji> Testing the disk too then? :)
[11:04] <db48x> yes
[11:05] <Senji> Any idea when shard5 comes out? :-)
[11:05] <db48x> gah, shard2 is broken the same way
[11:06] <db48x> weirdly it's exactly the same hash in the error messages...
[11:07] <db48x> oh man, git annex fsck is broken here too
[11:08] <db48x> it's just repeating this over and over:
[11:08] <db48x> git-annex: fd:12: commitBuffer: resource vanished (Broken pipe)
[11:08] <db48x> failed
[11:09] <Senji> Odd
[11:10] <db48x> indeed
[11:51] <iabak-reg> 03registrar 05master fc499f1 06other 10SHARD4/pubkeys registration of db48x iabak on SHARD4
[12:32] *** atomotic has joined #internetarchive.bak
[12:47] *** sankin has joined #internetarchive.bak
[13:16] *** primus104 has joined #internetarchive.bak
[13:40] <protodev> btw, i uploaded my current perl-script: https://github.com/cancerAlot/IA.BAK/tree/perl-master
[13:41] <protodev> it's still broken and not ready yet; if someone spots the bug in checkssh() which causes to register with an wrong ssh-pub-key, tell me ;)
[13:51] *** garyrh has quit IRC (http://bnc4free.com/)
[13:58] *** Start has quit IRC (Disconnected.)
[14:35] *** Start has joined #internetarchive.bak
[14:39] <db48x> protodev: excellent, I'll take a look
[14:59] *** Start has quit IRC (Disconnected.)
[15:00] <protodev> db48x: there is no maintain-function, no cronjob, no proper osx-support (in the install-sub) and loads of other missing features
[15:01] *** atomotic has quit IRC (Quit: Textual IRC Client: www.textualapp.com)
[15:03] <protodev> and the whole @cmd-thing will be replaced - ugly hack which is not needed
[15:03] *** Start has joined #internetarchive.bak
[15:16] *** Start has quit IRC (Disconnected.)
[15:31] <iabak-reg> 03registrar 05master b2119e2 06other 10SHARD3/pubkeys registration of protodev on SHARD3
[15:31] <protodev> YES! :D
[15:32] <protodev> it works !!
[15:32] <sep332> woot
[15:33] <protodev> ok, not yet...
[15:40] <iabak-reg> 03registrar 05master 106f19e 06other 10SHARD3/pubkeys registration of db48x+iabak on SHARD3
[15:41] <Senji> Hmm, the disk I have my shard4 repo in is nearly full; should I create a new shard4 repo on the next disk and worry about not having duplication; or just wait until shard5?
[15:42] <db48x> if you create another shard4, then you'll want to manually set them both up so that they don't want content that the other has
[15:43] <Senji> Yeah, exactly
[15:44] <sep332> can I do that with iabak or do i have to run "git-annex get" directly with appropriate flags?
[15:44] <db48x> use git annex wanted
[15:44] <iabak-reg> 03registrar 05master ed8d47f 06other 10SHARD3/pubkeys registration of protodev on SHARD3
[15:44] *** Start has joined #internetarchive.bak
[15:47] <Senji> The git annex wanted documentation says it doesn't have --in; which seems to rule out the obvious way of doing that (git annex wanted --not --in <uuid>)
[15:49] *** Start has quit IRC (Read error: Connection reset by peer)
[15:57] <db48x> yep, successfully registered
[15:57] <db48x> plenty of other problems after that though :)
[16:00] <db48x> protodev: I'll send you a pull request
[16:02] <db48x> it's probably similar to what you've already got, but since you said that yours didn't work...
[16:08] *** primus104 has quit IRC (Leaving.)
[16:16] <db48x> protodev: removed some duplicated bits
[16:18] <sep332> are more shards planned? just curious
[16:21] *** Start has joined #internetarchive.bak
[16:24] *** garyrh has joined #internetarchive.bak
[16:43] *** Start has quit IRC (Disconnected.)
[16:56] <protodev> db48x: we wrote the exactly same piece of code ^
[16:56] <protodev> i just named my variables a bit different
[17:23] *** primus104 has joined #internetarchive.bak
[17:26] <yipdw> the reincarnation of Thompson and Ritchie
[17:31] <db48x> :)
[17:31] <db48x> sep332: yes
[17:31] <db48x> sep332: the plan is to have enough shards to back up all of archive.org, of course
[17:32] <sep332> are you going to add each as the previous one gets close to done? or all done?
[17:32] <yipdw> we should call them crystals in preparation for Final Fantasy XVI or whatever
[17:33] <db48x> no, I believe the plan is to add all of the rest of them all at once, and spread the word wide and far
[17:34] <sep332> ok cool
[17:38] <db48x> I think there were a few more things we wanted to do before then, though
[17:44] *** primus104 has quit IRC (Leaving.)
[18:32] *** primus104 has joined #internetarchive.bak
[19:33] <tpw_rules> yipdw: you're only allowed to have 7 crystals
[19:33] <tpw_rules> perhaps make the number of total shards evenly divisible by 7?
[19:50] <closure> can make a couple more shards, just need to pick collections for them to get 100k-200k files
[19:51] <sep332> if the same file (by MD5) gets uploaded multiple times, will it get backed up more? or only 4 copies per MD5?
[19:52] <closure> if it's in the same shard, only once
[19:52] <closure> well, 4 times
[19:52] <Senji> closure: are there likely to be more shards soon then?
[19:53] <closure> once someone picks some collections
[19:53] <Senji> my shard4 disk is full, so if I want to start another one off it'd currently have to be another shard4 with some faff
[19:53] <closure> or I can just randomly pick some, shrug
[19:53] <Senji> Well, nearly full :)
[19:54] <sep332> hm. a file can be in multiple collections. If those collections are in different shards, the same file will count for both shards
[19:54] <closure> when a given filename is in multiple collections, it only gets into one shard
[19:54] <closure> but if the same file is in the IA under different names, it might end up redundantly in multiple shards
[19:55] <sep332> ok thanks
[19:55] <closure> (could sort and uniq this away, but it's a 32gb file listing them all..)
[19:55] <Senji> There are enough files in the IA that I'm sure there are MD5 collisions
[19:55] <closure> that too
[19:56] <closure> although iirc we did the math and collisions seemed unlikely
[19:56] <closure> at least collisions within a shard seem very unlikely
[19:56] <closure> (non-intentional collisions)
[19:56] <pikhq> As I understand it the risk of *accidental* MD5 collisions is still astronomically low.
[19:56] <sep332> MD5 collisions are extremely unlikely unless you control both files
[19:57] <pikhq> Or, "unless you are actually trying for a collision".
[19:57] <sep332> if i give you a static MD5, like one from the census, you couldn't make a file that collides.
[19:58] <sep332> there is a cryptanalytic weakness but it's still not good enough to be feasible
[20:02] <Senji> sep332: actually, people have created md5 collisions for existing other documents
[20:03] <Senji> sep332: but the thing here isn't malicious intent, it's the birthday 'paradox'
[20:07] <sep332> hadn't thought of birthday paradox. But I can't find any instance of preimage attack on md5?
[20:08] <closure> ./mkSHARD 'wwIIarchive dnalounge jstor_polisciequar usda-commoditysituationreports jstor_jpoliecon' 5
[20:08] <closure> fairly random, just a list an early shard packer found
[20:09] <Senji> sep332: the Flame malware forged a Windows code-signing cert
[20:18] <sep332> Senji2: that was a chosen-prefix collision, not a preimage. the attackers controlled part of the "original" certificate
[20:19] <Senji> sep332: oh, OK, my apologies; I must have misunderstood
[20:19] <Senji> I've seen a random collision (in work's backups) before :)
[20:19] <sep332> some questionable conclusions but a good summary https://randomoracle.wordpress.com/2012/06/13/unanswered-questions-about-the-flame-certificate-forgery-22/
[20:19] <sep332> oh, crazy
[20:29] <closure>  ./mkSHARD 'sports wikipediadumps Strangefolk canadianpamphlets statedocs_maine 1920_census' 6
[20:39] *** zottelbey has quit IRC (Remote host closed the connection)
[20:47] <closure> wow, this would be a BIG shard
[20:47] <closure> 6.19 terabytes
[20:47] <closure> I'll bet it's all the sports vids
[20:47] <Senji> Something to get teeth into :)
[20:47] <closure> maybe save that one for later
[20:48] <sep332> similar number of files though? ~100,000?
[20:48] <closure> yes
[20:48] <closure> hmm, sports is only 1.25 tb
[20:49] <closure> must be the wikipedia
[20:49] <closure> 1.44 tb
[20:50] <closure> everything in this shard is kinda large
[20:50] <closure> 1920 census is 2.6 tb
[20:51] *** sankin has quit IRC (Leaving.)
[20:51] <sep332> as in the year 1920? pretty sure they didn't have that many hard drives lo
[20:51] <sep332> lol
[20:52] <closure> yeah, damnifiknow what's in tha
[20:52] <sep332> high-res scans of every piece of paper?
[20:53] <closure> only 61 gb of canadian pamphlets
[20:54] <db48x> yes
[20:55] <db48x> scans of microfilm of photographs of the pieces of paper
[20:55] <closure> all the scratches lovingly preserved at 600dpi
[20:56] <sep332> a video of someone reading all the papers out loud
[20:57] <closure> sep332: a scanning tunneling microscophy of your hard drive, 300 yrs after you back this up
[20:57] <sep332> oh wait db48x are you serious?
[20:58] <db48x> yes
[20:59] <sep332> then "called it" ;)
[21:00] <db48x> closure: did you see the stuff about my broken shards from this morning?
[21:01] <closure> yeah, when a git repo is that hosed, you can use git-annex repair, but it will be slow on the shards. Probably better to re-clone and move in .git/config and .git/annex/objects
[21:02] <db48x> how is that different from fsck?
[21:02] <closure> fsck just finds problems
[21:02] <db48x> it fixed the first shard
[21:03] <closure> it doesn't fix broken git repos, it only fixes up git-annex specific issues
[21:05] <db48x> what do you think is broken?
[21:05] <closure> you have a bunch of broken .git/objects/ files
[21:05] <db48x> well, one
[21:06] <db48x> same hash repeated a number of times
[21:06] <db48x> what about the other message?
[21:06] <db48x> git-annex: fd:12: commitBuffer: resource vanished (Broken pipe)
[21:06] <db48x> failed
[21:07] <closure> just a consequence of the actual broken git repo
[21:37] <db48x> it'd be nice if it were more informative, and less spammy
[22:04] <iabak-reg> 03registrar 05master ad0b477 06other 10SHARD5/pubkeys registration of jdamery+iabak on SHARD5
[22:08] <iabak-reg> 03registrar 05master 1308265 06other 10SHARD3/pubkeys registration of protodev on SHARD3
[22:16] *** DFJustin has quit IRC (IMHOSTFU)
[22:17] *** DFJustin has joined #internetarchive.bak
[22:17] *** svchfoo2 sets mode: +o DFJustin
[22:17] *** svchfoo1 sets mode: +o DFJustin
[22:17] *** Start has joined #internetarchive.bak
[22:21] *** toad2 has joined #internetarchive.bak
[22:22] <iabak-reg> 03registrar 05master 67e3630 06other 10SHARD3/pubkeys registration of protodev on SHARD3
[22:27] *** toad1 has quit IRC (Read error: Operation timed out)
[23:30] <db48x> oh yea, this drive is toast:
[23:30] <db48x> [db48x@celebdil shard2]$ ../git-annex.linux/git-annex fsck --quiet
[23:30] <db48x> git-annex: .git/annex/objects/Gk/8X/MD5-s1843643129--f8d132c8f834ebf8597856bcfc069f8f/MD5-s1843643129--f8d132c8f834ebf8597856bcfc069f8f: hGetBufSome: hardware fault (Input/output error)
[23:31] <db48x> and also:
[23:31] <db48x> May 06 16:21:30 celebdil smartd[1196]: Device: /dev/sdc [SAT], 1062 Currently unreadable (pending) sectors
[23:31] <db48x> May 06 16:21:30 celebdil smartd[1196]: Device: /dev/sdc [SAT], 51 Offline uncorrectable sectors
[23:38] <Senji> loverly
[23:40] <db48x> actually, I suppose the word "toast" applies better to a different drive from a different computer which got burnt a couple of weeks ago
[23:40] <db48x> power supply shorted and caused a small fire
[23:44] <db48x> oh, even better:
[23:44] <db48x> May 06 15:16:16 celebdil kernel: EXT4-fs (sdc1): Delayed block allocation failed for inode 28312757 at logical offset 1162316 with max
[23:44] <db48x> May 06 15:16:16 celebdil kernel: EXT4-fs (sdc1): This should not happen!! Data will be lost
[23:44] <db48x> May 06 15:16:16 celebdil kernel: EXT4-fs error (device sdc1) in ext4_writepages:2395: Journal has aborted
[23:44] <db48x> I'm so glad I use ZFS for most things