| Time |
Nickname |
Message |
|
12:12
🔗
|
vrmlguy |
So, I like to peek at the URLs that I'm downloading. This week-end I found something both interesting and huge. Is there any way to clone the archive that's being built so I don't have to run my own wget? |
|
12:18
🔗
|
ersi |
What do you mean by cloning the archive? |
|
12:18
🔗
|
ersi |
Do you mean downloading all the data that others have downloaded? |
|
12:28
🔗
|
Smiley |
I think he just wants that single warc? |
|
12:30
🔗
|
ersi |
Ah. |
|
12:55
🔗
|
vrmlguy |
yeah, I just want a copy of what I'm downloading, to avoid duplicate work |
|
12:58
🔗
|
ersi |
The current scripts will remove what you've downloaded as soon as it's done with the uploading |
|
12:58
🔗
|
ersi |
but you could modify it so that it'll copy/ssh it somewhere before doing that I guess - or do it manually when it's in the rsync stage |
|
13:02
🔗
|
vrmlguy |
that sounds good. I guess I need to log into my warrior instead of treating it as a black box |
|
13:04
🔗
|
ersi |
could always find and download that warc later on though :) |
|
13:05
🔗
|
vrmlguy |
re-read "black box". where do the ward's wind up? |
|
13:05
🔗
|
vrmlguy |
warc |
|
13:06
🔗
|
ersi |
archive.org |
|
13:06
🔗
|
ersi |
sooner or later |
|
13:06
🔗
|
vrmlguy |
ah, of course. thanks |
|
13:07
🔗
|
vrmlguy |
- Downloaded 49840 URLs |
|
13:07
🔗
|
vrmlguy |
Starting WgetDownload for Item import-rkyd.posterous.com |
|
13:14
🔗
|
Smiley |
yah we hitting massive users :o |
|
20:17
🔗
|
alard |
Tip for vrmlguy (who's no longer here) and others: you can log in and make a hard-link to the warc file while it's downloading, so you can still get it when it's done. |
|
21:29
🔗
|
ersi |
I'll tip him off if he arrives again |