| Time |
Nickname |
Message |
|
00:00
🔗
|
Fusl |
i'm mering these two download directories now, fixing my fkup |
|
00:14
🔗
|
|
odemgi_ has quit IRC (Remote host closed the connection) |
|
00:14
🔗
|
|
sknebel has quit IRC (Read error: Operation timed out) |
|
00:14
🔗
|
|
wacky has quit IRC (Read error: Operation timed out) |
|
00:14
🔗
|
|
human has quit IRC (Remote host closed the connection) |
|
00:15
🔗
|
|
odemgi_ has joined #archiveteam-bs |
|
00:15
🔗
|
|
Dark_Star has quit IRC (Read error: Operation timed out) |
|
00:15
🔗
|
|
human has joined #archiveteam-bs |
|
00:15
🔗
|
|
bitBaron has quit IRC (Quit: My computer has gone to sleep. 😴😪ZZZzzz…) |
|
00:15
🔗
|
|
BlueMax has quit IRC (hub.dk efnet.port80.se) |
|
00:15
🔗
|
|
RichardG has quit IRC (hub.dk efnet.port80.se) |
|
00:15
🔗
|
|
brayden has quit IRC (hub.dk efnet.port80.se) |
|
00:15
🔗
|
|
Stiletto has quit IRC (hub.dk efnet.port80.se) |
|
00:15
🔗
|
|
lenary has quit IRC (hub.dk efnet.port80.se) |
|
00:15
🔗
|
|
omglolbah has quit IRC (hub.dk efnet.port80.se) |
|
00:15
🔗
|
|
LFlare has quit IRC (hub.dk efnet.port80.se) |
|
00:15
🔗
|
|
BartoCH has quit IRC (hub.dk efnet.port80.se) |
|
00:15
🔗
|
|
ReimuHaku has quit IRC (hub.dk efnet.port80.se) |
|
00:15
🔗
|
|
DFJustin has quit IRC (hub.dk efnet.port80.se) |
|
00:15
🔗
|
|
SketchCow has quit IRC (hub.dk efnet.port80.se) |
|
00:15
🔗
|
|
yuitimoth has quit IRC (hub.dk efnet.port80.se) |
|
00:15
🔗
|
|
arbin_ has quit IRC (hub.dk efnet.port80.se) |
|
00:15
🔗
|
|
Phoen1x_ has quit IRC (hub.dk efnet.port80.se) |
|
00:15
🔗
|
|
arkiver has quit IRC (hub.dk efnet.port80.se) |
|
00:15
🔗
|
|
zhongfu has quit IRC (hub.dk efnet.port80.se) |
|
00:15
🔗
|
|
sec^nd has quit IRC (hub.dk efnet.port80.se) |
|
00:15
🔗
|
|
Kenshin has quit IRC (hub.dk efnet.port80.se) |
|
00:15
🔗
|
|
decay has quit IRC (hub.dk efnet.port80.se) |
|
00:15
🔗
|
|
Jusque has quit IRC (hub.dk efnet.port80.se) |
|
00:15
🔗
|
|
acridAxid has quit IRC (hub.dk efnet.port80.se) |
|
00:15
🔗
|
|
tuluu has quit IRC (hub.dk efnet.port80.se) |
|
00:15
🔗
|
|
Sanqui has quit IRC (hub.dk efnet.port80.se) |
|
00:15
🔗
|
|
Jon has quit IRC (hub.dk efnet.port80.se) |
|
00:15
🔗
|
|
t3 has quit IRC (hub.dk efnet.port80.se) |
|
00:15
🔗
|
|
bsmith093 has quit IRC (hub.dk efnet.port80.se) |
|
00:15
🔗
|
|
Ing3b0rg has quit IRC (hub.dk efnet.port80.se) |
|
00:15
🔗
|
|
obskyr has quit IRC (hub.dk efnet.port80.se) |
|
00:15
🔗
|
|
gandalf has quit IRC (hub.dk efnet.port80.se) |
|
00:15
🔗
|
|
atbk has quit IRC (hub.dk efnet.port80.se) |
|
00:15
🔗
|
|
ephemer0l has quit IRC (hub.dk efnet.port80.se) |
|
00:15
🔗
|
|
nyaomi has quit IRC (hub.dk efnet.port80.se) |
|
00:15
🔗
|
|
jeekl has quit IRC (hub.dk efnet.port80.se) |
|
00:15
🔗
|
|
eientei95 has quit IRC (hub.dk efnet.port80.se) |
|
00:15
🔗
|
|
cynthia has quit IRC (hub.dk efnet.port80.se) |
|
00:15
🔗
|
|
apache2 has quit IRC (hub.dk efnet.port80.se) |
|
00:15
🔗
|
|
hook54321 has quit IRC (hub.dk efnet.port80.se) |
|
00:15
🔗
|
|
Zebranky has quit IRC (hub.dk efnet.port80.se) |
|
00:15
🔗
|
|
horkermon has quit IRC (hub.dk efnet.port80.se) |
|
00:15
🔗
|
|
colona has quit IRC (hub.dk efnet.port80.se) |
|
00:15
🔗
|
|
Fusl_ has quit IRC (hub.dk efnet.port80.se) |
|
00:15
🔗
|
|
revi has quit IRC (hub.dk efnet.port80.se) |
|
00:15
🔗
|
|
octarine has quit IRC (hub.dk efnet.port80.se) |
|
00:15
🔗
|
|
chr1sm has quit IRC (hub.dk efnet.port80.se) |
|
00:15
🔗
|
|
HCross has quit IRC (hub.dk efnet.port80.se) |
|
00:15
🔗
|
|
Vito` has quit IRC (hub.dk efnet.port80.se) |
|
00:15
🔗
|
|
bitspill has quit IRC (hub.dk efnet.port80.se) |
|
00:15
🔗
|
|
pnJay has quit IRC (hub.dk efnet.port80.se) |
|
00:15
🔗
|
|
riking has quit IRC (hub.dk efnet.port80.se) |
|
00:15
🔗
|
|
diggan has quit IRC (hub.dk efnet.port80.se) |
|
00:15
🔗
|
|
JSharp has quit IRC (hub.dk efnet.port80.se) |
|
00:15
🔗
|
|
tsr has quit IRC (hub.dk efnet.port80.se) |
|
00:15
🔗
|
|
bakJAA has quit IRC (hub.dk efnet.port80.se) |
|
00:15
🔗
|
|
ThisAsYou has quit IRC (hub.dk efnet.port80.se) |
|
00:15
🔗
|
|
Ctrl-S_ has quit IRC (hub.dk efnet.port80.se) |
|
00:15
🔗
|
|
deathy has quit IRC (hub.dk efnet.port80.se) |
|
00:15
🔗
|
|
DrasticAc has quit IRC (hub.dk efnet.port80.se) |
|
00:15
🔗
|
|
Meroje has quit IRC (hub.dk efnet.port80.se) |
|
00:15
🔗
|
|
kpcyrd has quit IRC (hub.dk efnet.port80.se) |
|
00:15
🔗
|
|
Hecatz has quit IRC (hub.dk efnet.port80.se) |
|
00:15
🔗
|
|
Muad-Dib has quit IRC (hub.dk efnet.port80.se) |
|
00:15
🔗
|
|
sknebel has joined #archiveteam-bs |
|
00:15
🔗
|
|
halt has quit IRC (Read error: Operation timed out) |
|
00:16
🔗
|
|
wacky has joined #archiveteam-bs |
|
00:16
🔗
|
|
bitBaron has joined #archiveteam-bs |
|
00:17
🔗
|
|
DarkStar1 has joined #archiveteam-bs |
|
00:17
🔗
|
|
apache2 has joined #archiveteam-bs |
|
00:17
🔗
|
|
BlueMax has joined #archiveteam-bs |
|
00:17
🔗
|
|
RichardG has joined #archiveteam-bs |
|
00:17
🔗
|
|
brayden has joined #archiveteam-bs |
|
00:17
🔗
|
|
lenary has joined #archiveteam-bs |
|
00:17
🔗
|
|
Stiletto has joined #archiveteam-bs |
|
00:17
🔗
|
|
omglolbah has joined #archiveteam-bs |
|
00:17
🔗
|
|
LFlare has joined #archiveteam-bs |
|
00:17
🔗
|
|
BartoCH has joined #archiveteam-bs |
|
00:17
🔗
|
|
ReimuHaku has joined #archiveteam-bs |
|
00:17
🔗
|
|
DFJustin has joined #archiveteam-bs |
|
00:17
🔗
|
|
SketchCow has joined #archiveteam-bs |
|
00:17
🔗
|
|
yuitimoth has joined #archiveteam-bs |
|
00:17
🔗
|
|
arbin_ has joined #archiveteam-bs |
|
00:17
🔗
|
|
arkiver has joined #archiveteam-bs |
|
00:17
🔗
|
|
sec^nd has joined #archiveteam-bs |
|
00:17
🔗
|
|
zhongfu has joined #archiveteam-bs |
|
00:17
🔗
|
|
Kenshin has joined #archiveteam-bs |
|
00:17
🔗
|
|
decay has joined #archiveteam-bs |
|
00:17
🔗
|
|
Jusque has joined #archiveteam-bs |
|
00:17
🔗
|
|
acridAxid has joined #archiveteam-bs |
|
00:17
🔗
|
|
tuluu has joined #archiveteam-bs |
|
00:17
🔗
|
|
Sanqui has joined #archiveteam-bs |
|
00:17
🔗
|
|
Jon has joined #archiveteam-bs |
|
00:17
🔗
|
|
t3 has joined #archiveteam-bs |
|
00:17
🔗
|
|
Ing3b0rg has joined #archiveteam-bs |
|
00:17
🔗
|
|
bsmith093 has joined #archiveteam-bs |
|
00:17
🔗
|
|
obskyr has joined #archiveteam-bs |
|
00:17
🔗
|
|
gandalf has joined #archiveteam-bs |
|
00:17
🔗
|
|
atbk has joined #archiveteam-bs |
|
00:17
🔗
|
|
cynthia has joined #archiveteam-bs |
|
00:17
🔗
|
|
ephemer0l has joined #archiveteam-bs |
|
00:17
🔗
|
|
Fusl_ has joined #archiveteam-bs |
|
00:17
🔗
|
|
nyaomi has joined #archiveteam-bs |
|
00:17
🔗
|
|
jeekl has joined #archiveteam-bs |
|
00:17
🔗
|
|
eientei95 has joined #archiveteam-bs |
|
00:17
🔗
|
|
efnet.port80.se sets mode: +oooo SketchCow arkiver Sanqui eientei95 |
|
00:17
🔗
|
|
hook54321 has joined #archiveteam-bs |
|
00:17
🔗
|
|
Zebranky has joined #archiveteam-bs |
|
00:17
🔗
|
|
horkermon has joined #archiveteam-bs |
|
00:17
🔗
|
|
revi has joined #archiveteam-bs |
|
00:17
🔗
|
|
colona has joined #archiveteam-bs |
|
00:17
🔗
|
|
octarine has joined #archiveteam-bs |
|
00:17
🔗
|
|
chr1sm has joined #archiveteam-bs |
|
00:17
🔗
|
|
HCross has joined #archiveteam-bs |
|
00:17
🔗
|
|
efnet.port80.se sets mode: +oo hook54321 HCross |
|
00:17
🔗
|
|
Vito` has joined #archiveteam-bs |
|
00:17
🔗
|
|
bitspill has joined #archiveteam-bs |
|
00:17
🔗
|
|
pnJay has joined #archiveteam-bs |
|
00:17
🔗
|
|
riking has joined #archiveteam-bs |
|
00:17
🔗
|
|
diggan has joined #archiveteam-bs |
|
00:17
🔗
|
|
JSharp has joined #archiveteam-bs |
|
00:17
🔗
|
|
tsr has joined #archiveteam-bs |
|
00:17
🔗
|
|
bakJAA has joined #archiveteam-bs |
|
00:17
🔗
|
|
ThisAsYou has joined #archiveteam-bs |
|
00:17
🔗
|
|
Ctrl-S_ has joined #archiveteam-bs |
|
00:17
🔗
|
|
deathy has joined #archiveteam-bs |
|
00:17
🔗
|
|
DrasticAc has joined #archiveteam-bs |
|
00:17
🔗
|
|
Meroje has joined #archiveteam-bs |
|
00:17
🔗
|
|
kpcyrd has joined #archiveteam-bs |
|
00:17
🔗
|
|
Hecatz has joined #archiveteam-bs |
|
00:17
🔗
|
|
Muad-Dib has joined #archiveteam-bs |
|
00:17
🔗
|
|
efnet.port80.se sets mode: +o bakJAA |
|
00:17
🔗
|
|
JAA sets mode: +o bakJAA |
|
00:17
🔗
|
|
Phoen1x has joined #archiveteam-bs |
|
00:18
🔗
|
|
halt has joined #archiveteam-bs |
|
00:18
🔗
|
|
bakJAA sets mode: +o JAA |
|
00:22
🔗
|
|
Sk1d has quit IRC (Read error: Operation timed out) |
|
00:26
🔗
|
|
Sk1d has joined #archiveteam-bs |
|
00:28
🔗
|
|
bitBaron has quit IRC (Quit: My computer has gone to sleep. 😴😪ZZZzzz…) |
|
00:34
🔗
|
|
bitBaron has joined #archiveteam-bs |
|
01:05
🔗
|
|
Sk1d has quit IRC (Read error: Operation timed out) |
|
01:08
🔗
|
|
Sk1d has joined #archiveteam-bs |
|
01:10
🔗
|
|
benjinsmi has quit IRC (Quit: Leaving) |
|
01:12
🔗
|
|
human has quit IRC (Remote host closed the connection) |
|
01:12
🔗
|
|
human has joined #archiveteam-bs |
|
01:13
🔗
|
|
atomicthu has quit IRC (Read error: Operation timed out) |
|
01:13
🔗
|
|
Fletcher has quit IRC (Read error: Operation timed out) |
|
01:14
🔗
|
|
halt has quit IRC (Read error: Operation timed out) |
|
01:15
🔗
|
|
atomicthu has joined #archiveteam-bs |
|
01:17
🔗
|
|
halt has joined #archiveteam-bs |
|
01:20
🔗
|
|
Fletcher has joined #archiveteam-bs |
|
01:25
🔗
|
|
Sk1d has quit IRC (Read error: Operation timed out) |
|
01:27
🔗
|
|
Sk1d has joined #archiveteam-bs |
|
01:50
🔗
|
|
Mateon1 has quit IRC (Ping timeout: 612 seconds) |
|
01:54
🔗
|
|
Mateon1 has joined #archiveteam-bs |
|
02:00
🔗
|
|
ndiddy has quit IRC () |
|
02:01
🔗
|
|
human has quit IRC (Remote host closed the connection) |
|
02:02
🔗
|
|
human has joined #archiveteam-bs |
|
02:04
🔗
|
|
human has quit IRC (Remote host closed the connection) |
|
02:38
🔗
|
znak |
mr_archiv: I'm pretty sure the YouTube API has a free quota the resets every day, but I never actually tried pushing it to the limit. And yeah, creating a Google account w/o a phone number can be hard. |
|
02:39
🔗
|
znak |
Unrelated to YouTube comments, Super Mario Maker 2 was just announced, and now there's the question of how long Nintendo will keep hosting Super Mario Maker 1 levels. There are at least hundreds of thousands of levels. |
|
02:40
🔗
|
znak |
I seem to remember some people reverse-engineering the data formats, but does anyone know of an effort to archive levels? |
|
02:41
🔗
|
znak |
The closest thing I know of is https://github.com/tachyo/SMMData, which is metadata (only) of 300K levels, scraped from the web service at supermariomakerbookmark.nintendo.net. |
|
02:47
🔗
|
|
Sk1d has quit IRC (Read error: Operation timed out) |
|
02:49
🔗
|
|
Sk1d has joined #archiveteam-bs |
|
02:51
🔗
|
|
halt has quit IRC (Read error: Operation timed out) |
|
02:53
🔗
|
|
halt has joined #archiveteam-bs |
|
02:59
🔗
|
|
Sk1d has quit IRC (Read error: Operation timed out) |
|
03:02
🔗
|
|
Sk1d has joined #archiveteam-bs |
|
03:24
🔗
|
|
Sk1d has quit IRC (Read error: Operation timed out) |
|
03:26
🔗
|
|
Sk1d has joined #archiveteam-bs |
|
03:32
🔗
|
|
Sk1d has quit IRC (Read error: Operation timed out) |
|
03:36
🔗
|
|
Sk1d has joined #archiveteam-bs |
|
03:50
🔗
|
|
Sk1d has quit IRC (Read error: Operation timed out) |
|
03:54
🔗
|
|
Sk1d has joined #archiveteam-bs |
|
04:00
🔗
|
|
glmd has joined #archiveteam-bs |
|
04:01
🔗
|
glmd |
Hey we're trying to archive Blogger comments. |
|
04:01
🔗
|
glmd |
The worker is here: https://github.com/afrmtbl/blogspot-comment-backup |
|
04:01
🔗
|
glmd |
We'd appreciate some help. |
|
04:04
🔗
|
|
glmd has quit IRC (Client Quit) |
|
04:07
🔗
|
Fusl |
JAA: 614 zips |
|
04:10
🔗
|
|
bitBaron has quit IRC (Quit: My computer has gone to sleep. 😴😪ZZZzzz…) |
|
04:14
🔗
|
|
odemgi has joined #archiveteam-bs |
|
04:15
🔗
|
|
bobmcjr has joined #archiveteam-bs |
|
04:17
🔗
|
bobmcjr |
I'm trying to archive a phpBB forum, but it's misconfigured so it doesn't comprehend cookies. The result is many duplicates due to the &sid= tacked on to the URL, and it seems that it might not finish. |
|
04:17
🔗
|
|
odemgi_ has quit IRC (Read error: Operation timed out) |
|
04:21
🔗
|
mr_archiv |
glmd, I am interested in this. Can you be more specific on what you need help with. |
|
04:24
🔗
|
bobmcjr |
glmd? |
|
04:25
🔗
|
mr_archiv |
bobmcjr, have you considered removing &sid from the URL before fetching them? |
|
04:26
🔗
|
bobmcjr |
phpBB autogenerates it as I crawl. It also seems to change |
|
04:26
🔗
|
mr_archiv |
urllib.parse.urlsplit then urllib.parse.parse_qs then urllib.parse.urlunsplit |
|
04:28
🔗
|
bobmcjr |
I've just been using wget --mirror --warc-file |
|
04:29
🔗
|
mr_archiv |
Can you post-process the URLs via stdout? |
|
04:30
🔗
|
mr_archiv |
I have the real answer. You need to use a session. |
|
04:30
🔗
|
mr_archiv |
Then the sid will stay the same I think. |
|
04:31
🔗
|
mr_archiv |
Wait nevermind, I am wrong about that. |
|
04:32
🔗
|
mr_archiv |
I don't understand why the sid would change, sid = session ID. |
|
04:32
🔗
|
bobmcjr |
I'm trying to tell if the duplicates are from my session, or from hyperlinks people have embedded that have their sids |
|
04:33
🔗
|
mr_archiv |
Good point, I did not think of people accidently leaking their session ID. |
|
04:33
🔗
|
mr_archiv |
Can you generate a list of URLs and save it to a file. |
|
04:34
🔗
|
mr_archiv |
Then post-process the file with a Python script then instruct wget to fetch all the URLs. |
|
04:34
🔗
|
mr_archiv |
After removing the session IDs you can use sort -u so they won't be fetched twice. |
|
04:35
🔗
|
mr_archiv |
Actually I just relized, there is no point in doing this. |
|
04:35
🔗
|
mr_archiv |
Depending on how this is done it will end up being harder on the server than just fetching the duplicates. |
|
04:36
🔗
|
mr_archiv |
I think the only way is to remove the session ID before attempting to fetch the URLs |
|
04:36
🔗
|
bobmcjr |
I'll try again and just hope my sid remains stable so that there are a finite number of URLs. |
|
04:37
🔗
|
bobmcjr |
This is the first thing I've tried to crawl for proper archiving |
|
04:37
🔗
|
mr_archiv |
Generally speaking you don't have to go through the kind of trouble like I described. Traditional web crawling works for most websites. |
|
04:37
🔗
|
mr_archiv |
I mean for archiving. |
|
04:38
🔗
|
bobmcjr |
For reference it's this, which may or may not disappear in 24-48 hours: forum.eltechs.com/index.php |
|
04:42
🔗
|
bobmcjr |
Ok, my sid seems stable. I'll just tolerate duplicates. |
|
04:53
🔗
|
|
qw3rty113 has joined #archiveteam-bs |
|
04:59
🔗
|
|
qw3rty112 has quit IRC (Read error: Operation timed out) |
|
04:59
🔗
|
|
godane has quit IRC (Remote host closed the connection) |
|
05:02
🔗
|
|
godane has joined #archiveteam-bs |
|
05:17
🔗
|
|
wp494 has quit IRC (Read error: Operation timed out) |
|
05:17
🔗
|
|
wp494 has joined #archiveteam-bs |
|
05:19
🔗
|
|
Sk1d has quit IRC (Read error: Operation timed out) |
|
05:22
🔗
|
|
Sk1d has joined #archiveteam-bs |
|
05:45
🔗
|
|
wp494 has quit IRC (Ping timeout: 252 seconds) |
|
05:47
🔗
|
|
wp494 has joined #archiveteam-bs |
|
06:16
🔗
|
bobmcjr |
Straight wget can't handle this. It is looping. Is there any more flexible software for crawling to assemble a WARC? |
|
06:28
🔗
|
jodizzle |
bobmcjr: Check out https://github.com/ludios/grab-site |
|
06:31
🔗
|
|
superkuh has joined #archiveteam-bs |
|
06:32
🔗
|
|
deevious has joined #archiveteam-bs |
|
07:06
🔗
|
|
Stilett0 has joined #archiveteam-bs |
|
07:16
🔗
|
|
Stiletto has quit IRC (Ping timeout: 615 seconds) |
|
07:47
🔗
|
|
wyatt8740 has quit IRC (Ping timeout: 360 seconds) |
|
07:48
🔗
|
|
Albardin has quit IRC (Read error: Operation timed out) |
|
07:48
🔗
|
|
paul2520 has quit IRC (Read error: Operation timed out) |
|
07:48
🔗
|
|
wacky has quit IRC (Write error: Broken pipe) |
|
07:48
🔗
|
|
kiska1 has quit IRC (Read error: Operation timed out) |
|
07:48
🔗
|
|
wacky has joined #archiveteam-bs |
|
07:50
🔗
|
|
PhrackD has quit IRC (Read error: Operation timed out) |
|
07:50
🔗
|
|
GLaDOS has quit IRC (Write error: Broken pipe) |
|
07:51
🔗
|
|
HashbangI has quit IRC (Read error: Operation timed out) |
|
07:51
🔗
|
|
sep332 has quit IRC (Read error: Operation timed out) |
|
07:51
🔗
|
|
dxrt_ has quit IRC (Read error: Operation timed out) |
|
07:51
🔗
|
|
qw3rty113 has quit IRC (Read error: Operation timed out) |
|
07:51
🔗
|
|
step has quit IRC (Read error: Operation timed out) |
|
07:53
🔗
|
|
qw3rty113 has joined #archiveteam-bs |
|
07:53
🔗
|
|
kiska1 has joined #archiveteam-bs |
|
07:53
🔗
|
|
Albardin has joined #archiveteam-bs |
|
07:55
🔗
|
|
kiska1 has quit IRC (Read error: Operation timed out) |
|
07:59
🔗
|
|
TigerbotH has quit IRC (Ping timeout: 600 seconds) |
|
07:59
🔗
|
|
bobmcjr has quit IRC (Read error: Operation timed out) |
|
07:59
🔗
|
|
Sk1d has quit IRC (Read error: Operation timed out) |
|
07:59
🔗
|
|
Albardin has quit IRC (Read error: Operation timed out) |
|
08:00
🔗
|
|
Albardin has joined #archiveteam-bs |
|
08:02
🔗
|
|
qw3rty114 has joined #archiveteam-bs |
|
08:02
🔗
|
|
kiska1 has joined #archiveteam-bs |
|
08:02
🔗
|
|
Sk1d has joined #archiveteam-bs |
|
08:03
🔗
|
|
bobmcjr has joined #archiveteam-bs |
|
08:03
🔗
|
|
paul2520 has joined #archiveteam-bs |
|
08:03
🔗
|
|
qw3rty113 has quit IRC (Ping timeout: 600 seconds) |
|
08:05
🔗
|
|
TigerbotH has joined #archiveteam-bs |
|
08:05
🔗
|
|
dxrt_ has joined #archiveteam-bs |
|
08:05
🔗
|
|
dxrt sets mode: +o dxrt_ |
|
08:08
🔗
|
|
PhrackD has joined #archiveteam-bs |
|
08:08
🔗
|
|
Exairnous has quit IRC (Ping timeout: 252 seconds) |
|
08:09
🔗
|
|
step has joined #archiveteam-bs |
|
08:09
🔗
|
|
sep332 has joined #archiveteam-bs |
|
08:09
🔗
|
|
HashbangI has joined #archiveteam-bs |
|
08:11
🔗
|
|
GLaDOS has joined #archiveteam-bs |
|
08:58
🔗
|
JAA |
Fusl: lol nice. Do you have a log file for a complete grab? (If not, I'll wget --spider it.) |
|
09:00
🔗
|
JAA |
bobmcjr: Interesting, I haven't seen that before. Yeah, it's going to be very tricky to archive that. You could ignore any URL which doesn't have the initial SID in it, but then you'd miss parts of the forums (it would stop recursing when the SID changes). Or you could strip out the SID, as wpull does (and hence ArchiveBot and grab-site), but then all links will be broken. :-/ There probably is no |
|
09:00
🔗
|
JAA |
good solution. |
|
09:01
🔗
|
JAA |
Try ignoring the "Delete all board cookies" in the footer at least though. That should help with the changing SIDs. |
|
09:02
🔗
|
JAA |
Or maybe you could go after URLs like http://forum.eltechs.com/viewtopic.php?p=3329 instead. Unbrowsable, but at least all the content would be there. |
|
09:04
🔗
|
JAA |
A two-phase approach might also work: first archive only the thread lists, then extract the thread URLs and grab those. No recursion, simple lists of URLs. That might even preserve browsability mostly. |
|
09:24
🔗
|
|
Phoen1x has quit IRC (Quit: Leaving) |
|
10:00
🔗
|
|
BlueMax has quit IRC (Quit: Leaving) |
|
10:01
🔗
|
|
Stilett0 has quit IRC (Read error: Operation timed out) |
|
10:02
🔗
|
|
Stiletto has joined #archiveteam-bs |
|
11:49
🔗
|
|
lindalap has joined #archiveteam-bs |
|
11:49
🔗
|
|
lindalap has quit IRC (Client Quit) |
|
12:26
🔗
|
|
bitBaron has joined #archiveteam-bs |
|
13:06
🔗
|
|
sep332 has quit IRC (ZNC 1.6.3+deb1ubuntu0.1 - http://znc.in) |
|
13:35
🔗
|
kiska |
urgh on the blox shutdown |
|
13:48
🔗
|
|
bitBaron has quit IRC (My computer has gone to sleep. 😴😪ZZZzzz…) |
|
14:02
🔗
|
|
Sk1d has quit IRC (Read error: Operation timed out) |
|
14:04
🔗
|
|
Sk1d has joined #archiveteam-bs |
|
14:07
🔗
|
PurpleSym |
JAA: There’s a blog search, making discovery fairly easy: http://www.blox.pl/html/2883585,13107202,81.html?y |
|
14:07
🔗
|
PurpleSym |
I’ll try to get a list of blogs. |
|
14:11
🔗
|
|
Sk1d has quit IRC (Read error: Operation timed out) |
|
14:14
🔗
|
Fusl |
JAA: nope i don't |
|
14:14
🔗
|
|
Sk1d has joined #archiveteam-bs |
|
14:45
🔗
|
JAA |
PurpleSym: Hmm, I found this list: http://www.blox.pl/html/65537.html |
|
14:45
🔗
|
JAA |
Also the category lists linked on the right on that page. |
|
14:48
🔗
|
PurpleSym |
There’s no pagination on the first one, so it can hardly be a complete list of blogs. |
|
14:49
🔗
|
|
wp494 has quit IRC (Ping timeout: 268 seconds) |
|
14:49
🔗
|
|
wp494 has joined #archiveteam-bs |
|
14:49
🔗
|
|
Sk1d has quit IRC (Read error: Operation timed out) |
|
14:50
🔗
|
PurpleSym |
Categories are looking better. I’ll scrape them as well. |
|
14:51
🔗
|
JAA |
There are tags also: http://www.blox.pl/blog/tags/index |
|
14:52
🔗
|
PurpleSym |
Thanks, I’ll see how close I can get to 689136 URLs with these lists. |
|
14:53
🔗
|
|
Sk1d has joined #archiveteam-bs |
|
14:53
🔗
|
JAA |
One more: http://www.blox.pl/blog/index |
|
14:54
🔗
|
JAA |
With the groups at the top ( |
|
14:54
🔗
|
PurpleSym |
Hm, or is it 665438 blogs? |
|
14:54
🔗
|
JAA |
"Polecamy", "Biznes", etc.) |
|
14:54
🔗
|
JAA |
665k blogs |
|
15:29
🔗
|
|
Fusl has quit IRC (Quit: oh god please send help) |
|
15:31
🔗
|
|
Fusl has joined #archiveteam-bs |
|
15:58
🔗
|
arkiver |
blox.pl sounds like a warrior project |
|
15:58
🔗
|
arkiver |
:) |
|
15:59
🔗
|
arkiver |
Can probably set something up tomorrow if it´s not too difficult to archive |
|
16:04
🔗
|
PurpleSym |
arkiver: I should have a first list ready by then. As for archiving: If wpull preserves cookies, you should ignore URLs like this one: http://xxx.blox.pl/html?mobile=yes |
|
16:43
🔗
|
|
bitBaron has joined #archiveteam-bs |
|
16:53
🔗
|
|
bitBaron has quit IRC (My computer has gone to sleep. 😴😪ZZZzzz…) |
|
16:59
🔗
|
|
bitBaron has joined #archiveteam-bs |
|
18:44
🔗
|
|
astrid has left ][ |
|
18:57
🔗
|
|
adwus has joined #archiveteam-bs |
|
19:08
🔗
|
|
Exairnous has joined #archiveteam-bs |
|
19:09
🔗
|
|
adwus has quit IRC (Ping timeout: 360 seconds) |
|
19:43
🔗
|
|
schbirid has joined #archiveteam-bs |
|
19:56
🔗
|
|
godane has quit IRC (Read error: Connection reset by peer) |
|
19:57
🔗
|
|
sep332 has joined #archiveteam-bs |
|
19:58
🔗
|
|
BartoCH has quit IRC (Ping timeout: 615 seconds) |
|
20:02
🔗
|
|
BartoCH has joined #archiveteam-bs |
|
20:14
🔗
|
|
godane has joined #archiveteam-bs |
|
20:22
🔗
|
|
BartoCH has quit IRC (Ping timeout: 615 seconds) |
|
20:32
🔗
|
|
fredgido has quit IRC (Read error: Connection reset by peer) |
|
20:36
🔗
|
|
m007a83_ has joined #archiveteam-bs |
|
20:39
🔗
|
|
m007a83 has quit IRC (Ping timeout: 252 seconds) |
|
20:48
🔗
|
|
bitBaron has quit IRC (My computer has gone to sleep. 😴😪ZZZzzz…) |
|
20:57
🔗
|
|
bitBaron has joined #archiveteam-bs |
|
21:06
🔗
|
|
BartoCH has joined #archiveteam-bs |
|
21:09
🔗
|
|
bitBaron has quit IRC (Quit: My computer has gone to sleep. 😴😪ZZZzzz…) |
|
21:09
🔗
|
|
bitBaron has joined #archiveteam-bs |
|
21:13
🔗
|
|
Despatche has quit IRC (Read error: Operation timed out) |
|
21:19
🔗
|
|
VerifiedJ has joined #archiveteam-bs |
|
21:21
🔗
|
|
VerifiedJ has quit IRC (Client Quit) |
|
21:22
🔗
|
|
VerifiedJ has joined #archiveteam-bs |
|
21:39
🔗
|
|
fredgido has joined #archiveteam-bs |
|
22:16
🔗
|
|
fredgido_ has joined #archiveteam-bs |
|
22:16
🔗
|
|
Albardin has quit IRC (Read error: Operation timed out) |
|
22:16
🔗
|
|
paul2520 has quit IRC (Read error: Operation timed out) |
|
22:16
🔗
|
|
dxrt_ has quit IRC (Write error: Broken pipe) |
|
22:16
🔗
|
|
kiska1 has quit IRC (Read error: Operation timed out) |
|
22:17
🔗
|
|
sep332 has quit IRC (Write error: Broken pipe) |
|
22:17
🔗
|
|
HashbangI has quit IRC (Read error: Operation timed out) |
|
22:18
🔗
|
|
PotcFdk has quit IRC (Read error: Operation timed out) |
|
22:18
🔗
|
|
step has quit IRC (Read error: Operation timed out) |
|
22:19
🔗
|
|
qw3rty114 has quit IRC (Read error: Operation timed out) |
|
22:21
🔗
|
|
Fusl has quit IRC (Read error: Operation timed out) |
|
22:21
🔗
|
|
PhrackD has quit IRC (Ping timeout: 600 seconds) |
|
22:22
🔗
|
|
Fusl has joined #archiveteam-bs |
|
22:22
🔗
|
|
Albardin has joined #archiveteam-bs |
|
22:22
🔗
|
|
kiska1 has joined #archiveteam-bs |
|
22:22
🔗
|
|
qw3rty114 has joined #archiveteam-bs |
|
22:22
🔗
|
|
fredgido has quit IRC (Read error: Operation timed out) |
|
22:22
🔗
|
|
TigerbotH has quit IRC (Read error: Connection reset by peer) |
|
22:23
🔗
|
|
paul2520 has joined #archiveteam-bs |
|
22:23
🔗
|
|
GLaDOS has quit IRC (Ping timeout: 600 seconds) |
|
22:25
🔗
|
|
GLaDOS has joined #archiveteam-bs |
|
22:25
🔗
|
|
PhrackD has joined #archiveteam-bs |
|
22:26
🔗
|
|
TigerbotH has joined #archiveteam-bs |
|
22:26
🔗
|
|
sep332 has joined #archiveteam-bs |
|
22:27
🔗
|
|
BlueMax has joined #archiveteam-bs |
|
22:28
🔗
|
|
HashbangI has joined #archiveteam-bs |
|
22:30
🔗
|
|
dxrt_ has joined #archiveteam-bs |
|
22:30
🔗
|
|
dxrt sets mode: +o dxrt_ |
|
22:30
🔗
|
|
step has joined #archiveteam-bs |
|
22:43
🔗
|
|
icedice has joined #archiveteam-bs |
|
23:11
🔗
|
|
PotcFdk has joined #archiveteam-bs |
|
23:13
🔗
|
|
schbirid has quit IRC (Remote host closed the connection) |
|
23:14
🔗
|
|
ndiddy has joined #archiveteam-bs |
|
23:48
🔗
|
|
Mateon1 has quit IRC (Read error: Operation timed out) |
|
23:48
🔗
|
|
Mateon1 has joined #archiveteam-bs |
|
23:50
🔗
|
|
wp494 has quit IRC (Read error: Operation timed out) |
|
23:50
🔗
|
|
icedice has quit IRC (Read error: Operation timed out) |
|
23:51
🔗
|
|
wp494 has joined #archiveteam-bs |
|
23:56
🔗
|
|
Sk1d has quit IRC (Read error: Operation timed out) |
|
23:59
🔗
|
|
Sk1d has joined #archiveteam-bs |