[00:07] *** niyaje3 has joined #internetarchive.bak [00:09] *** niyaje4 has joined #internetarchive.bak [00:16] *** niyaje3 has quit IRC (Read error: Operation timed out) [00:18] *** svchfoo2 has joined #internetarchive.bak [00:19] *** svchfoo1 sets mode: +o svchfoo2 [00:34] *** svchfoo2 has quit IRC (Quit: Closing) [00:34] *** svchfoo2 has joined #internetarchive.bak [00:35] *** svchfoo3 sets mode: +o svchfoo2 [00:47] *** niyaje4 has quit IRC (Ping timeout: 600 seconds) [01:01] *** niyaje4 has joined #internetarchive.bak [01:05] *** Start has joined #internetarchive.bak [01:05] *** Start has quit IRC (Client Quit) [01:06] *** Start has joined #internetarchive.bak [01:16] *** wp494 has quit IRC (Ping timeout: 740 seconds) [01:16] *** wp494_ has joined #internetarchive.bak [01:16] *** wp494_ is now known as wp494 [01:29] ppiixx: cute, I'll bet that's the xml index files, that don't have a known size, so it continues downloading them past the diskreserve setting [01:30] * closure fixes.. [01:30] *** tpw_rules has left Evil will always triumph, because good is dumb. [01:30] *** tpw_rules has joined #internetarchive.bak [02:07] *** niyaje4 has quit IRC (Ping timeout: 600 seconds) [02:36] *** niyaje4 has joined #internetarchive.bak [02:50] *** zottelbey has quit IRC (Remote host closed the connection) [03:32] *** niyaje4 has quit IRC (Ping timeout: 600 seconds) [04:09] *** DopefishJ is now known as DFJustin [04:10] *** svchfoo1 sets mode: +o DFJustin [04:54] *** SketchCow has quit IRC (Read error: Connection reset by peer) [04:54] *** chfoo has quit IRC (Ping timeout: 306 seconds) [04:55] *** Quile_ has joined #internetarchive.bak [04:56] *** espes___ has quit IRC (ny.us.hub irc.teksavvy.ca) [04:56] *** Quile has quit IRC (ny.us.hub irc.teksavvy.ca) [04:56] *** closure has quit IRC (ny.us.hub irc.teksavvy.ca) [04:56] *** espes___ has joined #internetarchive.bak [04:56] *** closure has joined #internetarchive.bak [04:56] *** irc.teksavvy.ca sets mode: +o closure [04:56] *** chfoo has joined #internetarchive.bak [04:58] *** espes___ has quit IRC (ircd.choopa.net irc.teksavvy.ca) [04:58] *** closure has quit IRC (ircd.choopa.net irc.teksavvy.ca) [05:03] *** espes__ has joined #internetarchive.bak [05:05] *** svchfoo3 has quit IRC (Quit: Closing) [05:06] *** svchfoo3 has joined #internetarchive.bak [05:07] *** svchfoo1 sets mode: +o svchfoo3 [05:20] *** closure has joined #internetarchive.bak [05:20] *** svchfoo1 sets mode: +o closure [06:06] *** raylee has joined #internetarchive.bak [06:56] *** yipdw_ has joined #internetarchive.bak [06:58] *** yipdw has quit IRC (Read error: Operation timed out) [07:15] closure: yeah exactly that [07:27] *** fenn has quit IRC (Read error: Operation timed out) [07:55] *** yipdw_ is now known as yipdw [07:56] *** svchfoo3 sets mode: +o yipdw [08:07] *** Infreq has joined #internetarchive.bak [09:46] *** zottelbey has joined #internetarchive.bak [14:43] *** VADemon has joined #internetarchive.bak [15:03] *** atomotic has joined #internetarchive.bak [16:16] *** atomotic has quit IRC (Quit: Textual IRC Client: www.textualapp.com) [16:55] *** Start has quit IRC (Read error: Connection reset by peer) [16:55] *** Start_ has joined #internetarchive.bak [19:49] *** SN4T14_ has joined #internetarchive.bak [19:55] *** SN4T14 has quit IRC (Ping timeout: 369 seconds) [20:17] *** Start_ is now known as Start [20:54] *** wp494 has quit IRC (LOUD UNNECESSARY QUIT MESSAGES) [20:58] *** wp494 has joined #internetarchive.bak [21:04] *** antomati_ is now known as antomatic [21:31] tpw_rules: hey, you're at the top of http://iabak.archiveteam.org/stats/SHARD1.expireleaderboard , can you please run iabak periodically? [21:31] what does that mean [21:31] i haven't installed iabak yet [21:31] well, you need to [21:32] also i think that may be me from when i erased my first archive [21:32] lemme check my id [21:32] I suspect one of them is, and one of them isn't [21:32] periodically running iabak is how we'll know which repos still exist [21:32] ok. how do i check my id? and what does iabak do? [21:33] https://archive.org/download/Ttscribe/Ttscribe_files.xml is still darked and preventing me from doing a complete get [21:34] I suppose that more files will dark from time to time, I'm not worrying about that [21:34] i'm running fsck [21:35] are you running sync after? Are you using the right version of git-annex? iabak takes care of that stuff [21:35] yeah and yeah. i meant this second [21:37] how often do i come up for expiry? i'll set a cron job [21:39] also root@katie, root@iashard-de-01, root@iashard-lax-01 [21:39] currently we're looking at 1 week, may change [21:39] is fsck --fast enough? [21:39] it's too much. iabak has a faster method [21:40] yeah i saw it [21:40] can i just run iabak in a screen and leave it alone? [21:41] if you touch IA.BAK/NOMORE, iabak won't check out any more shards at all, and will just do maintenance [21:42] it doesn't currently keep running forever though.. can run from cron job [21:42] ok [21:42] is it re-entrant from a cron-job [21:42] ym run from cron and from command line? [21:42] i mean like if i run it daily and it starts a download which takes > day [21:43] will it realize it's alrady running and exit [21:43] not currently [21:43] people like to run multiple ones to use more BW [21:44] we could have a iabak-cronjob that is safe that way, and avoids the more expensive stuff [21:44] ie, doesn't download more shards [21:44] oh yeah, ok. forgot it dodn't matter [21:45] could you add a 'destination' system to git annex? [21:45] so i can say "put x GB here and y GB there" and it will fill them up as much as possible [21:46] ideally stuffing with little files to get as close to the max as possible [21:46] cause i have a billion hard drives just lying around and i want to attach them to one system [21:46] that's accomplished by having different clones of the repo in different places [21:46] use lvm :P [21:46] i don't want them to duplicate files though [21:47] yipdw: the problem with that is that if one drive fails, everything goes down [21:47] zfs in raidz2/z3 mode [21:47] you can teach git-annex that a repo doesn't want files that are in another repo [21:48] or you can move files manually from one to another [21:48] but they're different sizes and that wastes space on redundancy [21:48] well [21:48] i just want this for the couple dozen laptop and desktop drives i have lying around to be useful (and also stress-test if i need one for something) [21:48] cd shard1; git remote add otherdrvie /other/drive/shard1 ; git annex move --to otherdrive [21:49] it sounds like there's higher-layer solutions available so that's fine, but I'm usually a fan of pushing the physical->logical mapping deeper into the stack so I don't have to care [21:49] yipdw: the problem is the physical map is fairly fluid [21:50] *** Quile_ has quit IRC (Read error: Operation timed out) [21:50] *** Quile has joined #internetarchive.bak [21:50] and a drive dying killing the entire thing would be a waste of bandwidth and time [21:51] closure: how often does the leaderboard update? sync is done [21:53] only once an hour [22:07] so it turns out i actually have spare hard drives out my butt. found four enclosures + drives in 10 minutes that i didn't know i had [22:14] ha [22:25] lol just 6TB lying around [22:27] * closure points to shard2 and shard3 [22:33] we should just install a cron job [22:39] EDITOR="cat cron.example >>$1" crontab -e [23:22] *** zottelbey has quit IRC (Remote host closed the connection) [23:55] tpw_rules: still at the top of http://iabak.archiveteam.org/stats/SHARD1.expireleaderboard