| Time |
Nickname |
Message |
|
09:48
🔗
|
jonbro_ |
heya! |
|
09:48
🔗
|
jonbro_ |
could anyone help me with a last minute archive project? |
|
09:49
🔗
|
jonbro_ |
a local pittsburgh message board is going down, and I am trying to wget it all before it dissapears. |
|
09:50
🔗
|
Schbirid |
step a: link the linky link |
|
09:50
🔗
|
jonbro_ |
nevertellmetheodds.org |
|
09:50
🔗
|
jonbro_ |
I don't have a warrior project, just looping over each thread |
|
09:51
🔗
|
jonbro_ |
supposedly he was going to take it down today, so I am not sure what the countdown clock actually looks like. |
|
09:52
🔗
|
Schbirid |
if the threads are enough, for i in {1..135000}; do wget -nv http://nevertellmetheodds.org/t.php?id=$i; done :D |
|
09:53
🔗
|
jonbro_ |
cool. |
|
09:53
🔗
|
jonbro_ |
thanks much! |
|
09:54
🔗
|
Schbirid |
that will not include the inline images though |
|
09:54
🔗
|
jonbro_ |
ah I am not worried about that, it is all offsite links. |
|
09:54
🔗
|
jonbro_ |
those were always inpermanent |
|
09:57
🔗
|
Schbirid |
for i in {102960..135000}; do wget -e robots=off -nv --page-requisites --span-hosts --reject-regex="(/favicon.ico|/style.css)" --exclude-domains=ajax.googleapis.com,www.businesscasualarchnemesis.com --convert-links --adjust-extension http://nevertellmetheodds.org/t.php?id=$i; done |
|
09:57
🔗
|
Schbirid |
small test, should grab inline images |
|
09:58
🔗
|
godane |
so i'm grabbing sitemap of funnyordie.com |
|
09:58
🔗
|
godane |
there is like 230 of them |
|
09:59
🔗
|
godane |
from there i can grab the video pages |
|
09:59
🔗
|
godane |
but i really don't have too |
|
09:59
🔗
|
jonbro_ |
hmmmm... |
|
10:00
🔗
|
godane |
the videos are hosted like this: http://vo.fod4.com/v/befceb53c6/v600.mp4 |
|
10:00
🔗
|
Schbirid |
i would recommend logging, add a "-a logfile_date.log". wget will print all messages there then |
|
10:00
🔗
|
godane |
the url is this: http://www.funnyordie.com/videos/befceb53c6/dani-weirdo-music-for-weirdos |
|
10:01
🔗
|
jonbro_ |
ok, will do :D got this running on two computers and an ec2 instance now :D |
|
10:02
🔗
|
Schbirid |
different number ranges, right? :) |
|
10:02
🔗
|
jonbro_ |
yep |
|
10:02
🔗
|
jonbro_ |
hopefully will finish before the owner wakes up and pulls the plug |
|
10:03
🔗
|
godane |
you need just the part of the url to get the video paths |
|
10:03
🔗
|
Schbirid |
:D |
|
10:03
🔗
|
godane |
also change video to v1200.mp4 so you get the high bit rate version |
|
10:05
🔗
|
Schbirid |
i started from 135k and 100k going down |
|
10:23
🔗
|
ivan` |
jonbro_: if you use wget --warc-file your archive could be put into wayback |
|
10:24
🔗
|
ivan` |
if there is no WARC, it's usually a lost cause |
|
10:25
🔗
|
jonbro_ |
oh really :( |
|
10:25
🔗
|
ivan` |
and starting one wget per URL is super dumb especially when you're grabbing page requisites |
|
10:25
🔗
|
jonbro_ |
can I warc it after the fact? |
|
10:25
🔗
|
ivan` |
but also because it doesn't reuse connections |
|
10:25
🔗
|
ivan` |
no |
|
10:25
🔗
|
jonbro_ |
oh rly???? |
|
10:25
🔗
|
jonbro_ |
oh shit. |
|
10:25
🔗
|
jonbro_ |
I mean, there is only one page requsite, just a style.css |
|
10:26
🔗
|
ivan` |
oh that's not so bad then |
|
10:26
🔗
|
jonbro_ |
ok, cool, thank god :D |
|
10:26
🔗
|
Schbirid |
ivan`: would you rather make an input file with urls? thought about that too |
|
10:27
🔗
|
ivan` |
that, or a terrible hack like this is what I'm using for one site |
|
10:27
🔗
|
ivan` |
end = (i+1)*10000 |
|
10:27
🔗
|
ivan` |
for i in range(0, 107): |
|
10:27
🔗
|
ivan` |
print start, end |
|
10:27
🔗
|
ivan` |
start = i*10000 + 1 |
|
10:27
🔗
|
ivan` |
os.system(r'wget --output-document=tmp --warc-file=bugzilla.redhat.com-%s-%s --warc-cdx -e robots=off https://bugzilla.redhat.com/show_bug.cgi\?id={%s..%s}' % (str(start).zfill(8), str(end).zfill(8), start, end)) |
|
10:27
🔗
|
jonbro_ |
ah interesting. |
|
10:28
🔗
|
jonbro_ |
is that faster? |
|
10:28
🔗
|
ivan` |
yes |
|
10:28
🔗
|
Schbirid |
oh wow yes |
|
10:28
🔗
|
jonbro_ |
ok, once one of these chunks end will switch over to that. |
|
10:29
🔗
|
Schbirid |
for i in {0..135000}; do echo "http://nevertellmetheodds.org/t.php?id=$i" >> urls; done |
|
10:29
🔗
|
Schbirid |
wget -nv --convert-links --adjust-extension --warc-file=nevertellmetheodds.org_20140301 --warc-cdx -i urls |
|
10:29
🔗
|
Schbirid |
several times as fast |
|
10:29
🔗
|
* |
Schbirid hides in a corner |
|
10:29
🔗
|
ivan` |
yes |
|
10:30
🔗
|
Cameron_D |
Just stick & after the wget call. Parallelise things :-) |
|
10:30
🔗
|
ivan` |
heh |
|
10:30
🔗
|
Schbirid |
"how to annoy a webserver" :P |
|
10:30
🔗
|
ivan` |
how to get a precious IPv4 IP banned for life |
|
10:30
🔗
|
ersi |
:D |
|
10:31
🔗
|
ersi |
is this #superhackysatursayday? |
|
10:31
🔗
|
Cameron_D |
Pretty sure you system would fall over first, trying to start 135k processes at once :P |
|
10:31
🔗
|
Smiley |
nah, i'll be fine |
|
10:31
🔗
|
Smiley |
you'll OOM pretty quickly tho |
|
10:31
🔗
|
Smiley |
the voice of experience ^^^ |
|
10:32
🔗
|
unbeholde |
schibirid I have a small problem |
|
10:32
🔗
|
Schbirid |
another unsatistied customer :(( |
|
10:33
🔗
|
unbeholde |
just found that stealth project alpha is corrupt and I tried to download it but it failed shortly before finishing. Same goes for Wonderful Life |
|
10:33
🔗
|
Schbirid |
give me the urls and i will re-grab them, might have been my fault |
|
10:34
🔗
|
jonbro_ |
can I make multiple warc files? or is that a no no if I want to chuck it on archive.org |
|
10:34
🔗
|
Cameron_D |
you can make multiple and then merge them, I beleive |
|
10:35
🔗
|
garyrh |
jonbro_, https://github.com/alard/megawarc |
|
10:36
🔗
|
jonbro_ |
cool! |
|
10:38
🔗
|
unbeholde |
what was that text upload service called? |
|
10:39
🔗
|
Schbirid |
pastee.org is nice |
|
10:39
🔗
|
unbeholde |
hmm ah here ok I got it up on pastebin http://pastebin.ca/2648419 |
|
10:40
🔗
|
Schbirid |
ok |
|
11:08
🔗
|
Schbirid |
unbeholde: https://www.quaddicted.com/files/temp/fp/ |
|
11:08
🔗
|
Schbirid |
7z says they are intact |
|
11:33
🔗
|
unbeholde |
ah thank you wonderfile life is working, just waiting for the stealth project to finish! By the way I'm having trouble uploading Ons2.0.zip and ut3domfinal_winsetup.zip at the moment, it failed on me twice. Do you happen to have a direct link for them? |
|
11:36
🔗
|
* |
unbeholde slaps Schbirid around a bit with a large fishbot |
|
11:36
🔗
|
Schbirid |
yay |
|
11:36
🔗
|
Schbirid |
uploading? |
|
11:37
🔗
|
unbeholde |
mmm to the almighty modDB |
|
11:37
🔗
|
Schbirid |
nothing more direct than the tarview links |
|
11:40
🔗
|
Nemo_bis |
Meh, why does a search for "rare chunks" only bring up academic papers? BitTorrent can't be more stupid than eMule, can it? I hope peers send out rare chunks first. https://encrypted.google.com/search?q=bittorrent+"rare+chunks" |
|
11:41
🔗
|
unbeholde |
it says tarview.php isn't a recognised file format |
|
11:42
🔗
|
Schbirid |
Nemo_bis: there is super seeding but apart from that iirc clients request what they want |
|
11:47
🔗
|
Nemo_bis |
Really? But clients don't even know the other leechers, they can only ask the wrong chunks... |
|
11:49
🔗
|
Schbirid |
https://wiki.theory.org/BitTorrentSpecification#Piece_downloading_strategy |
|
11:52
🔗
|
Nemo_bis |
Hm. https://trac.transmissionbt.com/ticket/3767 is marked fixed but I'm rather sure chrystal ball feature has not been implemented yet. |
|
11:55
🔗
|
Nemo_bis |
Dunno how much http://www.cs.sfu.ca/~mhefeeda/Papers/ism09.pdf applies |
|
16:30
🔗
|
Asparagir |
Need ops in ArchiveBot plz. Want to feed in Crimean sites, including media, while there's still time. |
|
23:30
🔗
|
gui77 |
given that myopera bebo and canvas are already at the max rate and push is (practically) finished, is there any other project needing bandwidth? :) |
|
23:55
🔗
|
sanqui |
hello. I know you folks are not archive.org, but I've been wondering if anybody can tell me if I'm fine uploading something there |
|
23:56
🔗
|
sanqui |
or if there's a better place for both short-term availability and long-term preservation |
|
23:57
🔗
|
sanqui |
basically, I don't know if you've heard about twitch plays pokémon, but it's been quite a phenomenon: https://en.wikipedia.org/wiki/Twitch_Plays_Pok%C3%A9mon |
|
23:57
🔗
|
sanqui |
I've got nearly 3gb of logs, and many people have asked me to provide them for academical study |
|
23:58
🔗
|
sanqui |
it's just under 400mb compressed, but I still don't want to host it myself |