[00:15] *** Start (~Start@[redacted]) has joined #internetarchive.bak
[00:16] *** svchfoo2 gives channel operator status to Start
[00:29] *** thunk (4746deec@[redacted]) has joined #internetarchive.bak
[00:40] *** londoncal (~londoncal@[redacted]) has joined #internetarchive.bak
[01:08] <closure> sep332: looks ok.. is there a way to construct an url from the fields there?
[01:08] <closure> also, I can't tell what if anything you've done to handle commas in filenames..
[01:09] *** thunk has quit (http://www.kiwiirc.com/ - A hand crafted IRC client)
[01:53] *** zottelbey has quit (Ping timeout: 512 seconds)
[02:06] <sep332> the URL is just https://archive.org/download/[itemid]/[filename]
[02:07] <sep332> would just putting quotes around the filenames help?
[02:08] <Ctrl-S> percent encode?
[02:10] *** thunk (4746deec@[redacted]) has joined #internetarchive.bak
[02:11] <sep332> that would be nicer but I don't know how to do that yet
[02:23] <Ctrl-S> wut language?
[02:24] <sep332> jq
[02:25] <sep332> http://stedolan.github.io/jq/manual/#Formatstringsandescaping
[02:25] <sep332> looks promising
[02:25] *** thunk has quit (http://www.kiwiirc.com/ - A hand crafted IRC client)
[02:29] *** marvinw has quit (Read error: Operation timed out)
[02:37] *** thunk (4746deec@[redacted]) has joined #internetarchive.bak
[02:38] *** phuzion has quit (Read error: Operation timed out)
[02:41] *** marvinw (~marvinw@[redacted]) has joined #internetarchive.bak
[02:46] *** johtso has quit (Quit: Connection closed for inactivity)
[02:50] *** niyaje has quit (Ping timeout: 600 seconds)
[03:21] *** phuzion (~phuzion@[redacted]) has joined #internetarchive.bak
[03:50] *** thunk has quit (http://www.kiwiirc.com/ - A hand crafted IRC client)
[04:42] *** db48x has quit (Read error: Connection reset by peer)
[04:44] <espes___> I'm kinda impressed jq is a 'language'
[05:22] <SketchCow> sep332: What's the conclusions? How much space in duplicates?
[07:49] *** thunk (4746deec@[redacted]) has joined #internetarchive.bak
[07:50] *** thunk has quit (Client Quit)
[08:27] *** X-Scale (~gbabios@[redacted]) has joined #internetarchive.bak
[09:02] *** GLaDOS (~STR_IDENT@[redacted]) has joined #internetarchive.bak
[09:10] *** serapeum (~serapeum@[redacted]) has joined #internetarchive.bak
[09:47] *** bzc6p__ (~bzc6p@[redacted]) has joined #internetarchive.bak
[09:47] *** mhazinsk has quit (hub.efnet.us irc.umich.edu)
[09:47] *** trs80 has quit (hub.efnet.us irc.umich.edu)
[09:52] *** bzc6p_ has quit (Ping timeout: 600 seconds)
[09:53] *** Laverne has quit (Read error: Operation timed out)
[09:53] *** chazchaz has quit (Read error: Operation timed out)
[09:53] *** svchfoo1 has quit (Read error: Operation timed out)
[09:53] *** aschmitz has quit (Read error: Operation timed out)
[09:54] *** shabble has quit (Read error: Operation timed out)
[09:54] *** shabble (~shabble@[redacted]) has joined #internetarchive.bak
[09:55] *** aschmitz (~aschmitz@[redacted]) has joined #internetarchive.bak
[09:59] *** swebb has quit (Ping timeout: 369 seconds)
[10:00] *** chazchaz (~chazchaz@[redacted]) has joined #internetarchive.bak
[10:00] *** Laverne (~Laverne@[redacted]) has joined #internetarchive.bak
[10:00] *** swebb (~swebb@[redacted]) has joined #internetarchive.bak
[10:02] *** svchfoo1 (~chfoo1@[redacted]) has joined #internetarchive.bak
[10:02] *** phuzion has quit (Read error: Operation timed out)
[10:02] *** sep332 has quit (Read error: Operation timed out)
[10:02] *** achip has quit (Read error: Operation timed out)
[10:02] *** svchfoo2 gives channel operator status to svchfoo1
[10:03] *** londonca_ (~londoncal@[redacted]) has joined #internetarchive.bak
[10:03] *** rossdylan has quit (Read error: Operation timed out)
[10:03] *** dirt has quit (Read error: Operation timed out)
[10:03] *** acridAxid has quit (Read error: Operation timed out)
[10:04] *** acridAxid (~acridAxid@[redacted]) has joined #internetarchive.bak
[10:04] *** svchfoo1 gives channel operator status to acridAxid
[10:04] *** GLaDOS has quit (Read error: Operation timed out)
[10:05] *** GLaDOS (~STR_IDENT@[redacted]) has joined #internetarchive.bak
[10:06] *** hatseflat (~hatseflat@[redacted]) has joined #internetarchive.bak
[10:08] *** londoncal has quit (Read error: Operation timed out)
[10:09] *** hatsefla1 has quit (Read error: Operation timed out)
[10:10] *** wp494 has quit (Quit: LOUD UNNECESSARY QUIT MESSAGES)
[10:10] *** Ctrl-S has quit (Read error: Operation timed out)
[10:10] *** bzc6p__ has quit (Read error: Operation timed out)
[10:13] *** sep332 (~sep332@[redacted]) has joined #internetarchive.bak
[10:13] *** dirt (james@[redacted]) has joined #internetarchive.bak
[10:13] *** achip (~thechip@[redacted]) has joined #internetarchive.bak
[10:13] *** svchfoo1 gives channel operator status to sep332
[10:13] *** bzc6p (~bzc6p@[redacted]) has joined #internetarchive.bak
[10:15] *** phuzion (~phuzion@[redacted]) has joined #internetarchive.bak
[10:16] *** acridAxid has quit (Quit: Quitting)
[10:17] *** Ctrl-S (~Ctrl-S@[redacted]) has joined #internetarchive.bak
[10:17] *** svchfoo1 gives channel operator status to Ctrl-S
[10:18] *** wp494 (~wickedpla@[redacted]) has joined #internetarchive.bak
[10:18] *** acridAxid (~acridAxid@[redacted]) has joined #internetarchive.bak
[10:19] *** svchfoo2 gives channel operator status to acridAxid
[10:19] *** rossdylan (~rossdylan@[redacted]) has joined #internetarchive.bak
[10:36] *** Kenshin has quit (Ping timeout: 258 seconds)
[10:37] *** irc.umich.edu gives channel operator status to trs80 mhazinsk
[10:37] *** mhazinsk (~matt@[redacted]) has joined #internetarchive.bak
[10:37] *** trs80 (trs80@[redacted]) has joined #internetarchive.bak
[10:53] *** Kenshin (~rurouni@[redacted]) has joined #internetarchive.bak
[10:54] *** svchfoo1 gives channel operator status to Kenshin
[12:06] *** trs80 has quit (Ping timeout: 186 seconds)
[12:08] *** trs80 (~trs80@[redacted]) has joined #internetarchive.bak
[12:11] *** zottelbey (~zottelbey@[redacted]) has joined #internetarchive.bak
[12:41] *** X-Scale has quit (Ping timeout: 240 seconds)
[12:45] *** X-Scale (~gbabios@[redacted]) has joined #internetarchive.bak
[13:49] *** thunk (4746deec@[redacted]) has joined #internetarchive.bak
[14:17] <sep332> SketchCow: there are 659 TB of files listed. So more then half of that is duplicated.
[14:57] *** thunk has quit (http://www.kiwiirc.com/ - A hand crafted IRC client)
[15:20] *** destrudo has quit (Remote host closed the connection)
[15:31] *** destrudo (~destrudo@[redacted]) has joined #internetarchive.bak
[15:35] *** thunk (4746deec@[redacted]) has joined #internetarchive.bak
[15:39] *** johtso (uid563@[redacted]) has joined #internetarchive.bak
[16:10] *** thunk has quit (http://www.kiwiirc.com/ - A hand crafted IRC client)
[17:00] *** garyrh has quit (http://bnc4free.com/)
[17:20] *** thunk (4746deec@[redacted]) has joined #internetarchive.bak
[18:04] *** londonca_ has quit (Quit: Leaving...)
[18:05] *** londoncal (~londoncal@[redacted]) has joined #internetarchive.bak
[19:02] *** thunk has quit (http://www.kiwiirc.com/ - A hand crafted IRC client)
[19:49] *** thunk (4746deec@[redacted]) has joined #internetarchive.bak
[20:54] *** db48x (~user@[redacted]) has joined #internetarchive.bak
[20:55] *** garyrh_ (~Mithrandi@[redacted]) has joined #internetarchive.bak
[21:30] *** DFJustin has quit (Ping timeout: 740 seconds)
[21:42] *** thunk has quit (http://www.kiwiirc.com/ - A hand crafted IRC client)
[21:43] <closure> <SketchCow> Anyway, tracey says use md5sum AND sha-1
[21:44] <closure> still only md5sum in the current census
[21:44] <closure> getting sha1s into it would be really helpful.
[21:45] <closure> git-annex has only supported using md5 since february
[22:00] <closure> jq is what the IA guys are using for this dataset. It's pretty rad!
[22:00] <closure> zcat public-file-size-md_20150304205357.json.gz | ./jq '.files[] | .name, .size' |head
[22:00] <closure> "Sabeeluna-al-jeehed.mp3"
[22:00] <closure> 3227238
[22:00] <closure> "Dilon_kay_Hukmaran.mp3"
[22:00] <closure> 3737995
[22:01] <closure> sep332: I think I'll go with this, makes it easy to parse and deal with any changes in the censues
[22:07] <closure> jq --raw-output '.collection , .id , (.files[] | .name, .size)'
[22:11] *** bzc6p_ (~bzc6p@[redacted]) has joined #internetarchive.bak
[22:11] <closure> jq --raw-output '.collection , "i:" + .id , (.files[] | "f:" + .name , .size)'
[22:14] <closure>  jq --raw-output '.collection , "i:" + .id , (.files[] | "f:" + .name , "s:" + (.size | tostring))'
[22:18] *** bzc6p has quit (Read error: Operation timed out)
[22:44] *** thunk (4746deec@[redacted]) has joined #internetarchive.bak
[22:47] <closure> jq --raw-output '.id as $foo | .files[] | @uri "https://archive.org/\($foo)/\(.name)"' <-- this is pretty awesome, it's a list of urls!
[22:52] <db48x> :)
[22:56] <closure> kinda weird this 1st item in the census is 404 https://archive.org/details/Urdu-Trana-001
[22:57] *** rossdylan has quit (Read error: Operation timed out)
[23:20] <closure> argh, this dataset sometimes has id: "string" and sometimes the id is an array. makes it hard to get urls that always work
[23:43] <closure> jq --raw-output '(.collection[]? // .collection) as $coll | (.id[]? // .id) as $id | .files[] | @uri "\($coll)\thttps://archive.org/\($id)/\(.name)"'
[23:44] <closure> yay, that works for both arrays and not-arrays!
[23:45] <closure> jq --raw-output '(.collection[]? // .collection) as $coll | (.id[]? // .id) as $id | .files[] | @uri "\(.md5)\t\($coll)\thttps://archive.org/\($id)/\(.name)"'
[23:46] *** closure is running that now on the full census to get a md5_collection_url.txt
[23:46] <closure> already 3 million lines in the file, this is fast!