[02:09] I wonder if undersco2 could grep a bunch of .warc.gz's on IA [02:16] I think we'll just pull them down, heh [03:05] * joepie91 can't wait for 4TB disk [03:05] * joepie91 will finally be able to just download and process crap without worrying about space [03:14] does anyone have digitalocean droplets in SF or willing to spawn 5 of them for a half a day? [03:14] I will play human tracker and give you commands that grep megawarcs on IA [03:32] $1,204,860 in donations to internet archive [03:32] excellent [03:32] Time to fill these fucking new hard drives [03:33] enough to download 0.01% more of YouTube ;) [03:34] will there be enough SF breeze for the new racks? [07:51] joepie91: hahahaha, you say that now [07:55] anyone have a VPS in the US that can do 500GB inbound over the next 13-15 hours? [07:55] it is for grepping wretch.cc data that's already on archive.org [08:02] no disk space is required [08:03] unless you're down to your last 10MB in which case you have bigger problems, heh [11:23] ivan`: if it's just about running a script and nobody else came up, I may [11:38] Nemo_bis: PMed you with instructions [11:38] we have many more terabytes of megawarcs to do tomorrow [11:40] if you have no pv, you can remove it from the shell pipelines [15:39] http://i.imgur.com/dJcslBc.gif [15:46] WHAT FORSOOTH, PRITHEE TELL ME THE SECRET WORD [15:55] seems to not be answering - secret word is yahoosucks [15:55] Thankyou [16:33] Did we have a bot answering that? [16:36] there was at one time [16:38] Regarding processing, ivan` - I'm sad to say, that while FOS has been an excellent dumping ground for "oh shit oh shit get it somewhere", the disk performance and i/o has been rather disappointing. [16:39] And we're now so good at having banks of warriors slamming through places, that we can fill the disk in no time, even with the automatic pump going [16:39] And that's assuming FOS is doing nothing else, which as a dumping ground it constantly is. [16:39] Maybe I should ask the archive for another box just to be a pump and processing center. [16:39] sumppump [16:39] We can then use it for archivebot and other projects that are constantly spewing things [16:54] I just requested the bounce machine [16:54] bounce or sumppump [16:54] We'll have a couple accounts on there, running stuff for the purposes of pumping - variations of alard's script, and some yipdw specials, etc. [16:56] neat [16:57] Yeah. [16:57] And you guys can run some nice geeky performance checks on it [16:58] mostly, I'd just like to have access so I can occasionally tap in and see what's coming back [16:58] I haven't seen much of ArchiveBot's output since testing [17:12] Just decided to see how the godane inbox is doing and HOLY CRAP [17:12] so, sorting those right now. [17:13] i will need direct access to Believer's Voice of Victory collection: https://archive.org/search.php?query=creator%3A%22Kenneth%20Copeland%20Ministries%22%20AND%20%28collection%3Agodaneinbox%29 [17:14] 109 items and thats only the tip of it [17:20] maxed out? http://teamarchive0.fnf.archive.org:8088/mrtg/diskv3.html [17:22] please tell me fnf is supposed to stand for "fast and furious" [17:22] because that'd be awesome [17:28] https://archive.org/stream/Desert-Magazine-1967-10#page/n11/mode/2up [17:28] Most uncomfortable car model ever [17:35] that last 4 cds of 2000 of Game.exe don't have a nrg format image [17:36] they are in iso format [17:36] some isos in 2001 and 2002 are still in nrg format [17:36] and the rest are standard iso format [17:37] just thought you should know [19:43] does it make any sense to run wretch in the warrior? [19:43] i mean the website is shutdown isnt it [19:49] Dovahkiin: it's still up on the IPs we're grabbing from [19:50] /join #shipwretched [19:50] darned space :( [20:15] Regarding http://teamarchive0.fnf.archive.org:8088/mrtg/diskv3.html [20:16] That is very much the "convert to megaWARC, then upload, then repeat" [20:16] So it is jamming up and down [20:16] Making 50gb megawarcs turns out to really tax the system [20:17] I suppose there's no easy tweak to make that smoother? [20:17] One way would be to have the packing be on a second disk. [20:18] But the disks both have a lot of projects on them, so they're near full [20:23] The CPU doesn't seem too busy. Increasing gzip compression *might* reduce disk usage a tiny bit, who knows [20:23] Or it could just make things slower disks fuller :) [20:25] Nemo_bis: the easy tweak would be to use SSDs everywhere :P [20:25] but that's very expensive [20:26] we've got a megawarc packer running on an SSD for wretch/yahoo blog and it's pretty sweet [20:26] though the machine that's running the packer is also getting hit hard with other stuff [20:26] heh [22:17] Most uncomfortable car model ever <-- http://www.flamingmayo.com/firstchurchofpacman/pacrod.jpg [22:19] ok that's worse [23:18] [23:20] DFJustin: most awesome car ever! [23:20] happy newyear and stuff [23:22] Happy New year from the CET timezone!