| Time |
Nickname |
Message |
|
02:09
🔗
|
ivan` |
I wonder if undersco2 could grep a bunch of .warc.gz's on IA |
|
02:16
🔗
|
ivan` |
I think we'll just pull them down, heh |
|
03:05
🔗
|
* |
joepie91 can't wait for 4TB disk |
|
03:05
🔗
|
* |
joepie91 will finally be able to just download and process crap without worrying about space |
|
03:14
🔗
|
ivan` |
does anyone have digitalocean droplets in SF or willing to spawn 5 of them for a half a day? |
|
03:14
🔗
|
ivan` |
I will play human tracker and give you commands that grep megawarcs on IA |
|
03:32
🔗
|
SketchCow |
$1,204,860 in donations to internet archive |
|
03:32
🔗
|
ivan` |
excellent |
|
03:32
🔗
|
SketchCow |
Time to fill these fucking new hard drives |
|
03:33
🔗
|
ivan` |
enough to download 0.01% more of YouTube ;) |
|
03:34
🔗
|
ivan` |
will there be enough SF breeze for the new racks? |
|
07:51
🔗
|
xmc |
joepie91: hahahaha, you say that now |
|
07:55
🔗
|
ivan` |
anyone have a VPS in the US that can do 500GB inbound over the next 13-15 hours? |
|
07:55
🔗
|
ivan` |
it is for grepping wretch.cc data that's already on archive.org |
|
08:02
🔗
|
ivan` |
no disk space is required |
|
08:03
🔗
|
ivan` |
unless you're down to your last 10MB in which case you have bigger problems, heh |
|
11:23
🔗
|
Nemo_bis |
ivan`: if it's just about running a script and nobody else came up, I may |
|
11:38
🔗
|
ivan` |
Nemo_bis: PMed you with instructions |
|
11:38
🔗
|
ivan` |
we have many more terabytes of megawarcs to do tomorrow |
|
11:40
🔗
|
ivan` |
if you have no pv, you can remove it from the shell pipelines |
|
15:39
🔗
|
SketchCow |
http://i.imgur.com/dJcslBc.gif |
|
15:46
🔗
|
superbisk |
WHAT FORSOOTH, PRITHEE TELL ME THE SECRET WORD |
|
15:55
🔗
|
BiggieJon |
seems to not be answering - secret word is yahoosucks |
|
15:55
🔗
|
superbisk |
Thankyou <BiggieJon> |
|
16:33
🔗
|
SketchCow |
Did we have a bot answering that? |
|
16:36
🔗
|
BiggieJon |
there was at one time |
|
16:38
🔗
|
SketchCow |
Regarding processing, ivan` - I'm sad to say, that while FOS has been an excellent dumping ground for "oh shit oh shit get it somewhere", the disk performance and i/o has been rather disappointing. |
|
16:39
🔗
|
SketchCow |
And we're now so good at having banks of warriors slamming through places, that we can fill the disk in no time, even with the automatic pump going |
|
16:39
🔗
|
SketchCow |
And that's assuming FOS is doing nothing else, which as a dumping ground it constantly is. |
|
16:39
🔗
|
SketchCow |
Maybe I should ask the archive for another box just to be a pump and processing center. |
|
16:39
🔗
|
SketchCow |
sumppump |
|
16:39
🔗
|
SketchCow |
We can then use it for archivebot and other projects that are constantly spewing things |
|
16:54
🔗
|
SketchCow |
I just requested the bounce machine |
|
16:54
🔗
|
SketchCow |
bounce or sumppump |
|
16:54
🔗
|
SketchCow |
We'll have a couple accounts on there, running stuff for the purposes of pumping - variations of alard's script, and some yipdw specials, etc. |
|
16:56
🔗
|
yipdw |
neat |
|
16:57
🔗
|
SketchCow |
Yeah. |
|
16:57
🔗
|
SketchCow |
And you guys can run some nice geeky performance checks on it |
|
16:58
🔗
|
yipdw |
mostly, I'd just like to have access so I can occasionally tap in and see what's coming back |
|
16:58
🔗
|
yipdw |
I haven't seen much of ArchiveBot's output since testing |
|
17:12
🔗
|
SketchCow |
Just decided to see how the godane inbox is doing and HOLY CRAP |
|
17:12
🔗
|
SketchCow |
so, sorting those right now. |
|
17:13
🔗
|
godane |
i will need direct access to Believer's Voice of Victory collection: https://archive.org/search.php?query=creator%3A%22Kenneth%20Copeland%20Ministries%22%20AND%20%28collection%3Agodaneinbox%29 |
|
17:14
🔗
|
godane |
109 items and thats only the tip of it |
|
17:20
🔗
|
Nemo_bis |
maxed out? http://teamarchive0.fnf.archive.org:8088/mrtg/diskv3.html |
|
17:22
🔗
|
yipdw |
please tell me fnf is supposed to stand for "fast and furious" |
|
17:22
🔗
|
yipdw |
because that'd be awesome |
|
17:28
🔗
|
SketchCow |
https://archive.org/stream/Desert-Magazine-1967-10#page/n11/mode/2up |
|
17:28
🔗
|
SketchCow |
Most uncomfortable car model ever |
|
17:35
🔗
|
godane |
that last 4 cds of 2000 of Game.exe don't have a nrg format image |
|
17:36
🔗
|
godane |
they are in iso format |
|
17:36
🔗
|
godane |
some isos in 2001 and 2002 are still in nrg format |
|
17:36
🔗
|
godane |
and the rest are standard iso format |
|
17:37
🔗
|
godane |
just thought you should know |
|
19:43
🔗
|
Dovahkiin |
does it make any sense to run wretch in the warrior? |
|
19:43
🔗
|
Dovahkiin |
i mean the website is shutdown isnt it |
|
19:49
🔗
|
ivan` |
Dovahkiin: it's still up on the IPs we're grabbing from |
|
19:50
🔗
|
kyan |
/join #shipwretched |
|
19:50
🔗
|
kyan |
darned space :( |
|
20:15
🔗
|
SketchCow |
Regarding http://teamarchive0.fnf.archive.org:8088/mrtg/diskv3.html |
|
20:16
🔗
|
SketchCow |
That is very much the "convert to megaWARC, then upload, then repeat" |
|
20:16
🔗
|
SketchCow |
So it is jamming up and down |
|
20:16
🔗
|
SketchCow |
Making 50gb megawarcs turns out to really tax the system |
|
20:17
🔗
|
Nemo_bis |
I suppose there's no easy tweak to make that smoother? |
|
20:17
🔗
|
SketchCow |
One way would be to have the packing be on a second disk. |
|
20:18
🔗
|
SketchCow |
But the disks both have a lot of projects on them, so they're near full |
|
20:23
🔗
|
Nemo_bis |
The CPU doesn't seem too busy. Increasing gzip compression *might* reduce disk usage a tiny bit, who knows |
|
20:23
🔗
|
Nemo_bis |
Or it could just make things slower disks fuller :) |
|
20:25
🔗
|
yipdw |
Nemo_bis: the easy tweak would be to use SSDs everywhere :P |
|
20:25
🔗
|
yipdw |
but that's very expensive |
|
20:26
🔗
|
yipdw |
we've got a megawarc packer running on an SSD for wretch/yahoo blog and it's pretty sweet |
|
20:26
🔗
|
yipdw |
though the machine that's running the packer is also getting hit hard with other stuff |
|
20:26
🔗
|
Nemo_bis |
heh |
|
22:17
🔗
|
DFJustin |
<SketchCow> Most uncomfortable car model ever <-- http://www.flamingmayo.com/firstchurchofpacman/pacrod.jpg |
|
22:19
🔗
|
Nemo_bis |
ok that's worse |
|
23:18
🔗
|
m1das |
|
|
23:20
🔗
|
m1das |
DFJustin: most awesome car ever! |
|
23:20
🔗
|
m1das |
happy newyear and stuff |
|
23:22
🔗
|
norbert79 |
Happy New year from the CET timezone! |