[00:08] *** Start has joined #internetarchive.bak [00:20] *** primus104 has quit IRC (Leaving.) [00:21] *** Senji has quit IRC (Read error: Operation timed out) [00:37] *** Senji has joined #internetarchive.bak [00:47] I'm seeing repair! [00:48] closure: Should we be recruiting more storage? [01:56] I need to add some shards, need more good collections to put in them [03:52] *** SketchCow has quit IRC (Ping timeout: 240 seconds) [04:37] *** Senji has quit IRC (Read error: Operation timed out) [04:41] *** Senji has joined #internetarchive.bak [04:58] *** frontward is now known as Erkan [06:35] *** primus104 has joined #internetarchive.bak [07:08] *** SketchCow has joined #internetarchive.bak [08:02] *** Start has quit IRC (Read error: Connection reset by peer) [08:03] *** Start has joined #internetarchive.bak [08:46] *** primus104 has quit IRC (Leaving.) [09:46] *** atomotic has joined #internetarchive.bak [09:57] *** primus104 has joined #internetarchive.bak [10:27] *** zhongfu has quit IRC (Quit: Goodbye.) [10:28] *** zhongfu has joined #internetarchive.bak [10:44] *** atomotic has quit IRC (Quit: My Mac has gone to sleep. ZZZzzz…) [11:04] *** primus104 has quit IRC (Leaving.) [11:05] *** Senji has quit IRC (Read error: Operation timed out) [11:16] *** Senji has joined #internetarchive.bak [11:27] *** db48x has quit IRC (Read error: Connection reset by peer) [11:28] *** Senji has quit IRC (Ping timeout: 252 seconds) [11:30] *** atomotic has joined #internetarchive.bak [11:35] *** Senji has joined #internetarchive.bak [11:37] *** db48x` has joined #internetarchive.bak [11:39] *** db48x has joined #internetarchive.bak [11:41] *** db48x` has quit IRC (Client Quit) [11:51] *** Senji has quit IRC (Ping timeout: 252 seconds) [11:55] *** primus104 has joined #internetarchive.bak [12:09] *** Senji has joined #internetarchive.bak [12:22] *** atomotic has quit IRC (Quit: Textual IRC Client: www.textualapp.com) [12:31] *** Senji has quit IRC (Ping timeout: 506 seconds) [12:42] *** cloudmons has quit IRC (Ping timeout: 492 seconds) [12:51] *** Senji has joined #internetarchive.bak [13:07] *** atomotic has joined #internetarchive.bak [13:36] *** cloudmons has joined #internetarchive.bak [13:43] *** primus104 has quit IRC (Leaving.) [13:51] *** Start has quit IRC (Quit: Disconnected.) [14:22] *** atomotic has quit IRC (Quit: Textual IRC Client: www.textualapp.com) [14:38] *** Start has joined #internetarchive.bak [15:15] *** primus104 has joined #internetarchive.bak [15:40] *** primus104 has quit IRC (Leaving.) [15:51] *** Ash___ is now known as Ash [15:51] *** higgins_ has joined #internetarchive.bak [15:52] *** higgins_ is now known as chrishigg [15:57] Hey, folks. So my first 5TB disk is full (yay!). When this occurs, do I simply add a second disk and start a second instance of ia.bak running? (For context, I have lots of disk lying around, but typically in 2-5TB chunks.) [16:01] yep [16:03] *** Start has quit IRC (Quit: Disconnected.) [16:04] you'll also need to manually adjust the cron job (or systemd unit) so that the shards in the new location are fscked properly [16:05] and you'll want to ensure that you're either not downloading the same shards in both locations, or that your shards are ignoring files already in the other copy [16:08] makes sense. does the readme (or similar documentation) tell me where I can include/exclude shards? in my first run, I just kinda let it go nuts, didn't touch any config per se. [16:11] of course not, that would be crazy [16:11] you can edit the repolist file [16:12] you can change the mode to anything you like [16:12] it'll ignore any mode it doesn't recognize [16:12] *** Start has joined #internetarchive.bak [16:28] *** DFJustin has quit IRC (Remote host closed the connection) [16:28] *** DFJustin has joined #internetarchive.bak [16:57] *** primus104 has joined #internetarchive.bak [16:58] Oh right, the repolist file. OK. Thank you! [17:01] https://docs.google.com/spreadsheets/d/1kPZbkiFD_SBf8kJV-fYAu6xzTBuVKAAxzld6grnDUSU/edit?usp=sharing [17:01] To help with finding potential collections, here's a list of every collection we've done so far, sorted for easy finding. [17:02] chrishigg: you're welcome :) [17:15] *** Start has quit IRC (Quit: Disconnected.) [17:25] hmm, I dunno if we need to bother users about not downloading the same shard onto their multiple disks [17:26] I mean, it would not be ideal if someone ended up being the only one that had some of the files in a given shard, but otoh it adds a lot of complexity to avoid it, and it's not fully avoidable anyhow [17:27] SketchCow: IIRC db48x had an idea of a SHARD0 that contained a list like that, and other index information [17:32] oh yeah, we already have that collection list, automatically, here http://iabak.archiveteam.org/stats/ALL.collections (and, my shard generaion scripts even check it, yay) [17:35] Good. [17:55] *** beardicus has quit IRC (Quit: bye now) [18:00] *** beardicus has joined #internetarchive.bak [19:49] *** Start has joined #internetarchive.bak [19:56] *** Start has quit IRC (Quit: Disconnected.) [20:11] *** Start has joined #internetarchive.bak [20:20] *** Start has quit IRC (Quit: Disconnected.) [21:50] *** zz_CyberJ is now known as CyberJaco [22:00] *** garyrh has quit IRC (Quit: http://bnc4free.com/) [22:09] *** chrishigg has quit IRC (hub.se efnet.port80.se) [22:09] *** zhongfu has quit IRC (hub.se efnet.port80.se) [22:09] *** pikhq has quit IRC (hub.se efnet.port80.se) [22:09] *** lhobas has quit IRC (hub.se efnet.port80.se) [22:09] *** GLaDOS has quit IRC (hub.se efnet.port80.se) [22:09] *** jbenet__ has quit IRC (hub.se efnet.port80.se) [22:09] *** Ctrl-S has quit IRC (hub.se efnet.port80.se) [22:09] *** gamingrob has quit IRC (hub.se efnet.port80.se) [22:09] *** bpye has quit IRC (hub.se efnet.port80.se) [22:09] *** ppiixx has quit IRC (hub.se efnet.port80.se) [22:09] *** mattl has quit IRC (hub.se efnet.port80.se) [22:09] *** Muad-Dib has quit IRC (hub.se efnet.port80.se) [23:02] *** Start has joined #internetarchive.bak