[12:12] <vrmlguy> So, I like to peek at the URLs that I'm downloading.  This week-end I found something both interesting and huge.  Is there any way to clone the archive that's being built so I don't have to run my own wget?
[12:18] <ersi> What do you mean by cloning the archive?
[12:18] <ersi> Do you mean downloading all the data that others have downloaded?
[12:28] <Smiley> I think he just wants that single warc?
[12:30] <ersi> Ah.
[12:55] <vrmlguy> yeah, I just want a copy of what I'm downloading, to avoid duplicate work
[12:58] <ersi> The current scripts will remove what you've downloaded as soon as it's done with the uploading
[12:58] <ersi> but you could modify it so that it'll copy/ssh it somewhere before doing that I guess - or do it manually when it's in the rsync stage
[13:02] <vrmlguy> that sounds good.  I guess I need to log into my warrior instead of treating it as a black box
[13:04] <ersi> could always find and download that warc later on though :)
[13:05] <vrmlguy> re-read "black box".  where do the ward's wind up?
[13:05] <vrmlguy> warc
[13:06] <ersi> archive.org
[13:06] <ersi> sooner or later
[13:06] <vrmlguy> ah, of course.  thanks
[13:07] <vrmlguy> - Downloaded 49840 URLs
[13:07] <vrmlguy> Starting WgetDownload for Item import-rkyd.posterous.com
[13:14] <Smiley> yah we hitting massive users :o
[20:17] <alard> Tip for vrmlguy (who's no longer here) and others: you can log in and make a hard-link to the warc file while it's downloading, so you can still get it when it's done.
[21:29] <ersi> I'll tip him off if he arrives again