[00:00] *** kyan has quit IRC (Read error: Connection reset by peer) [00:04] how about https://archive.org/details/ephemera [00:05] that+78rpm will be 2 tb [00:07] moved shard generation to the iabak server from fos, and it runs about 10x as fast no [00:15] hmm, I can toss in oldtimeradio too, for a 2.9 tb shard [00:37] just when I'm out of disk space [00:40] closure: do you have a second for a generic git-annex question? [00:40] sure [00:40] is there a way I can annex a url and have it transformed after downloading? [00:41] I want to annex a gzipped file and have it end up ungzipped automatically [00:42] ah.. this is possible to do using the external special remote interface, but that may be overkill or not suited to what you're trying to do [00:42] git annex addurl https://archive.org/download/emularity_engine_jsmess/messa2600.js.gz --file emulators/jsmess/messa2600.js.gz [00:43] there's a example one for bittorrent (before it got built into git-annex); for *.torrent urls, it makes addurl add not the torrent file, but download the contents and add those [00:44] hmm [00:44] http://git-annex.branchable.com/special_remotes/external/git-annex-remote-torrent [00:44] if I did that, how would another user of the same repository get the file? [00:44] they have to install the program and enable the special remote, and then they can get files using it [00:45] figures [00:45] that's a bit unwieldy [00:45] for gz, yes.. it might be useful for tars or zips [00:45] should be able to do git annex addurl http://example.com/foo.gz --pipe gunzip --file foo [00:46] or something [00:47] *** niyaje4 has quit IRC (Ping timeout: 600 seconds) [00:48] could meticulously translate all shell syntax into command-line arguments and let us build arbitrarily complex scripts which get stored in-line... [00:48] or not :P [00:49] heh [00:51] maybe a smudge filter? [01:08] sounds more like it [01:43] ooh. so i'm "expired", at least on shard1. [01:43] but i'm running an iabak [01:44] those expires haven't happened yet [01:44] but, which repo is it? [01:45] oh, nm. i think both my iabaks were hung. [01:45] restarted [01:45] bert@storage is me [01:46] see if the uuid there matches your repo [01:46] it might be an old repo you had, if you deleted it.. the one that it wants to expire is not recorded as containing any files [01:48] oh ok. that would be old then. [01:48] what's the incantation for getting into the shell where i can run git annex info or what-have-you? [01:49] cd shard1; git config annex.uuid [01:49] runshell is what i was thinking of... but clearly that's not what i actually needed :) [01:50] ah, git-annex.linux/runshell [01:50] ok. that indeed does not match my current uuid. [01:50] great, expiry working as intended [01:53] https://archive.org/download/Ttscribe/Ttscribe_files.xml is still darked [01:54] i can't complete my IAdex unless that gets removed from shard1 [01:55] closure: can you fix that? i just synced [01:58] or amn i doing something wrong? [02:03] closure: ^ [02:10] *** kyan has joined #internetarchive.bak [02:20] *** cloudmons has quit IRC (Read error: Operation timed out) [02:21] *** cloudmons has joined #internetarchive.bak [02:57] *** SN4T14__ has quit IRC (Read error: Connection reset by peer) [02:58] *** cloudmons has quit IRC (Read error: Operation timed out) [03:00] *** SN4T14 has joined #internetarchive.bak [03:00] *** cloudmons has joined #internetarchive.bak [03:01] *** chfoo has quit IRC (Read error: Connection reset by peer) [03:14] *** beardicus has quit IRC (Quit: Sleep.) [03:16] *** chfoo has joined #internetarchive.bak [03:16] *** svchfoo1 sets mode: +o chfoo [04:08] *** berndj has quit IRC (Read error: Operation timed out) [04:36] *** zottelbey has joined #internetarchive.bak [05:03] *** zottelbey has quit IRC (Remote host closed the connection) [06:10] *** cloudmons has quit IRC (Read error: Operation timed out) [06:10] *** chfoo has quit IRC (ircd.choopa.net hub.efnet.us) [06:10] *** iten has quit IRC (ircd.choopa.net hub.efnet.us) [06:10] *** espes___ has quit IRC (ircd.choopa.net hub.efnet.us) [06:10] *** Quile has quit IRC (ircd.choopa.net hub.efnet.us) [06:10] *** SketchCow has quit IRC (ircd.choopa.net hub.efnet.us) [06:10] *** closure has quit IRC (ircd.choopa.net hub.efnet.us) [06:10] *** ersi has quit IRC (ircd.choopa.net hub.efnet.us) [06:10] *** bpye_ has quit IRC (ircd.choopa.net hub.efnet.us) [06:10] *** serapeum has quit IRC (ircd.choopa.net hub.efnet.us) [06:10] *** svchfoo3 has quit IRC (ircd.choopa.net hub.efnet.us) [06:10] *** destrudo has quit IRC (ircd.choopa.net hub.efnet.us) [06:10] *** Cameron_D has quit IRC (ircd.choopa.net hub.efnet.us) [06:10] *** balrog has quit IRC (ircd.choopa.net hub.efnet.us) [06:10] *** marvinw has quit IRC (ircd.choopa.net hub.efnet.us) [06:10] *** GLaDOS has quit IRC (ircd.choopa.net hub.efnet.us) [06:10] *** Lord_Nigh has quit IRC (ircd.choopa.net hub.efnet.us) [06:10] *** ppiixx has quit IRC (ircd.choopa.net hub.efnet.us) [06:10] *** hatseflat has quit IRC (ircd.choopa.net hub.efnet.us) [06:10] *** Muad-Dib has quit IRC (ircd.choopa.net hub.efnet.us) [06:10] *** Vito` has quit IRC (ircd.choopa.net hub.efnet.us) [06:10] *** lhobas has quit IRC (ircd.choopa.net hub.efnet.us) [06:10] *** mrfoo has quit IRC (ircd.choopa.net hub.efnet.us) [06:10] *** jbenet_ has quit IRC (ircd.choopa.net hub.efnet.us) [06:10] *** ryang has quit IRC (ircd.choopa.net hub.efnet.us) [06:10] *** antomatic has quit IRC (ircd.choopa.net hub.efnet.us) [06:10] *** Kazzy has quit IRC (ircd.choopa.net hub.efnet.us) [06:10] *** edsu_ has quit IRC (ircd.choopa.net hub.efnet.us) [06:10] *** chfoo- has quit IRC (ircd.choopa.net hub.efnet.us) [06:10] *** hater has quit IRC (ircd.choopa.net hub.efnet.us) [06:10] *** garyrh has quit IRC (ircd.choopa.net hub.efnet.us) [06:10] *** Senji has quit IRC (ircd.choopa.net hub.efnet.us) [06:10] *** raylee has quit IRC (ircd.choopa.net hub.efnet.us) [06:10] *** svchfoo2 has quit IRC (ircd.choopa.net hub.efnet.us) [06:10] *** Atluxity has quit IRC (ircd.choopa.net hub.efnet.us) [06:10] *** kyan has quit IRC (ircd.choopa.net hub.efnet.us) [06:10] *** db48x has quit IRC (ircd.choopa.net hub.efnet.us) [06:10] *** dirt has quit IRC (ircd.choopa.net hub.efnet.us) [06:10] *** midas has quit IRC (ircd.choopa.net hub.efnet.us) [06:10] *** patrickod has quit IRC (ircd.choopa.net hub.efnet.us) [06:10] *** Kenshin has quit IRC (ircd.choopa.net hub.efnet.us) [06:10] *** trs80 has quit IRC (ircd.choopa.net hub.efnet.us) [06:10] *** mhazinsk has quit IRC (ircd.choopa.net hub.efnet.us) [06:10] *** swebb has quit IRC (ircd.choopa.net hub.efnet.us) [06:10] *** fenn has quit IRC (ircd.choopa.net hub.efnet.us) [06:10] *** SN4T14 has quit IRC (ircd.choopa.net hub.efnet.us) [06:10] *** arkiver has quit IRC (ircd.choopa.net hub.efnet.us) [06:10] *** yipdw has quit IRC (ircd.choopa.net hub.efnet.us) [06:10] *** realeyes has quit IRC (ircd.choopa.net hub.efnet.us) [06:10] *** svchfoo1 has quit IRC (ircd.choopa.net hub.efnet.us) [06:10] *** chazchaz has quit IRC (ircd.choopa.net hub.efnet.us) [06:12] *** cloudmons has joined #internetarchive.bak [06:35] *** berndj has joined #internetarchive.bak [06:35] *** chfoo has joined #internetarchive.bak [06:35] *** SN4T14 has joined #internetarchive.bak [06:35] *** kyan has joined #internetarchive.bak [06:35] *** garyrh has joined #internetarchive.bak [06:35] *** raylee has joined #internetarchive.bak [06:35] *** svchfoo2 has joined #internetarchive.bak [06:35] *** Atluxity has joined #internetarchive.bak [06:35] *** balrog has joined #internetarchive.bak [06:35] *** db48x has joined #internetarchive.bak [06:35] *** ny.us.hub sets mode: +oooo chfoo garyrh svchfoo2 db48x [06:35] *** iten has joined #internetarchive.bak [06:35] *** Vito` has joined #internetarchive.bak [06:35] *** marvinw has joined #internetarchive.bak [06:35] *** GLaDOS has joined #internetarchive.bak [06:35] *** arkiver has joined #internetarchive.bak [06:35] *** Lord_Nigh has joined #internetarchive.bak [06:35] *** dirt has joined #internetarchive.bak [06:35] *** mhazinsk has joined #internetarchive.bak [06:35] *** trs80 has joined #internetarchive.bak [06:35] *** chazchaz has joined #internetarchive.bak [06:35] *** fenn has joined #internetarchive.bak [06:35] *** swebb has joined #internetarchive.bak [06:35] *** svchfoo1 has joined #internetarchive.bak [06:35] *** ny.us.hub sets mode: +oooo GLaDOS arkiver mhazinsk svchfoo1 [06:35] *** realeyes has joined #internetarchive.bak [06:35] *** Kenshin has joined #internetarchive.bak [06:35] *** yipdw has joined #internetarchive.bak [06:35] *** patrickod has joined #internetarchive.bak [06:35] *** midas has joined #internetarchive.bak [06:35] *** chfoo- has joined #internetarchive.bak [06:35] *** ppiixx has joined #internetarchive.bak [06:35] *** Senji has joined #internetarchive.bak [06:35] *** ersi has joined #internetarchive.bak [06:35] *** hatseflat has joined #internetarchive.bak [06:35] *** Muad-Dib has joined #internetarchive.bak [06:35] *** edsu_ has joined #internetarchive.bak [06:35] *** jbenet_ has joined #internetarchive.bak [06:35] *** mrfoo has joined #internetarchive.bak [06:35] *** lhobas has joined #internetarchive.bak [06:35] *** bpye_ has joined #internetarchive.bak [06:35] *** ryang has joined #internetarchive.bak [06:35] *** serapeum has joined #internetarchive.bak [06:35] *** antomatic has joined #internetarchive.bak [06:35] *** svchfoo3 has joined #internetarchive.bak [06:35] *** ny.us.hub sets mode: +oooo Kenshin yipdw ersi svchfoo3 [06:35] *** destrudo has joined #internetarchive.bak [06:35] *** Cameron_D has joined #internetarchive.bak [06:35] *** espes___ has joined #internetarchive.bak [06:35] *** closure has joined #internetarchive.bak [06:35] *** SketchCow has joined #internetarchive.bak [06:35] *** Quile has joined #internetarchive.bak [06:35] *** hater has joined #internetarchive.bak [06:35] *** Kazzy has joined #internetarchive.bak [06:35] *** ny.us.hub sets mode: +ooo closure SketchCow Kazzy [07:31] *** atomotic has joined #internetarchive.bak [09:23] *** niyaje4 has joined #internetarchive.bak [10:13] *** VADemon has joined #internetarchive.bak [10:18] *** niyaje4 has quit IRC (Ping timeout: 600 seconds) [10:43] *** atomotic has quit IRC (Quit: Textual IRC Client: www.textualapp.com) [11:19] *** beardicus has joined #internetarchive.bak [11:40] *** atomotic has joined #internetarchive.bak [11:44] *** zottelbey has joined #internetarchive.bak [12:48] *** sankin has joined #internetarchive.bak [13:02] *** VADemon has quit IRC (Read error: Connection reset by peer) [13:14] *** sankin has quit IRC (Leaving.) [13:26] *** sankin has joined #internetarchive.bak [13:26] freed up some space, so starting multiple iabaks. closure, could you maybe skip the fsck if it's been done recently? [13:26] touch a stamp file or something [13:56] *** Start has quit IRC (Disconnected.) [14:33] *** Start has joined #internetarchive.bak [14:44] fast fsck should be really fast now [14:44] Oh? [14:45] yea, seconds [14:47] do a git pull in IA.BAK to make sure you have the latest version of the scripts [14:52] The new shard begins [14:57] *** atomotic has quit IRC (Quit: Textual IRC Client: www.textualapp.com) [14:58] *** Start has quit IRC (Disconnected.) [15:02] *** Start has joined #internetarchive.bak [15:39] woop woop [15:51] *** Start has quit IRC (Disconnected.) [15:58] *** Start has joined #internetarchive.bak [16:38] *** Start has quit IRC (Disconnected.) [16:50] *** Start has joined #internetarchive.bak [17:42] *** Start has quit IRC (Disconnected.) [18:08] What's the amount of files per shard? [18:09] *** zottelbey has quit IRC (Remote host closed the connection) [18:11] SketchCow: 100kish [18:11] *** zottelbey has joined #internetarchive.bak [18:16] Thanks, that helps. [18:17] yw [19:29] *** Start has joined #internetarchive.bak [19:49] *** SN4T14_ has joined #internetarchive.bak [19:55] *** SN4T14 has quit IRC (Ping timeout: 369 seconds) [20:19] *** Start has quit IRC (Disconnected.) [20:53] *** sankin has quit IRC (Leaving.) [22:01] *** niyaje4 has joined #internetarchive.bak [22:04] *** zottelbey has quit IRC (Remote host closed the connection) [22:08] *** garyrh has quit IRC (Ping timeout: 506 seconds) [22:17] *** Start has joined #internetarchive.bak [22:17] *** svchfoo1 sets mode: +o Start [22:30] *** ersi has quit IRC (Ping timeout: 512 seconds) [22:31] *** ohhdemgir has joined #internetarchive.bak [22:49] db48x: I have 1dc6c53578949c922116decb24c6af417f323da6 switch fast fask to be a truely fast expiry-preventing ping and the shard1 "This shard is in maintenance mode; checking it." has taken 3 minutes so far [22:50] looking at maint(), it doesn't even call fastfsck [22:51] it does take a lock at least, so if I'd started them one after another, the second one would have skipped it [22:52] *** niyaje4 has quit IRC (Ping timeout: 600 seconds) [22:55] trs80: ah, indeed. that's a normal fsck [22:59] at least it took less than 10 minutes [23:00] Checking for any files that still need to be downloaded... is a bit slow too [23:01] *** Start has quit IRC (Read error: Connection reset by peer) [23:01] *** Start_ has joined #internetarchive.bak [23:05] again, about 10 minutes. and now for the shuf delay [23:06] I guess the real answer is a long running/parallel process to amortise these startup costs [23:48] *** Start_ has quit IRC (Read error: Connection reset by peer) [23:48] *** Start has joined #internetarchive.bak [23:49] *** svchfoo1 sets mode: +o Start