| Time |
Nickname |
Message |
|
06:43
🔗
|
Nemo_bis |
Archival of wiki-site.com at this speed (360 s wait between requests) will take about 880 days.... |
|
06:44
🔗
|
Nemo_bis |
Suggestions for other wait times welcome |
|
07:14
🔗
|
odie5533 |
361 s |
|
07:14
🔗
|
Nemo_bis |
https://constantsun.fogbugz.com/default.asp?183_5n6491ub |
|
07:14
🔗
|
odie5533 |
What is that fogbugz for? |
|
07:27
🔗
|
Nemo_bis |
EditThis bug tracker apparently |
|
07:57
🔗
|
w0rp |
I should upload my tanasinn.info archive later. I got the whole thing with dumpgenerator. |
|
07:58
🔗
|
w0rp |
(Because imageboard and BBS in-jokes must be preserved forever.) |
|
09:39
🔗
|
Nemo_bis |
good |
|
18:34
🔗
|
w0rp |
Okay, I filled out all of the forms on archive.org to add tansinn.info. |
|
18:35
🔗
|
w0rp |
I think it was a little too hard to do, and the forms could definitely use some improvement. I felt like I was applying for insurance or something while doing it. |
|
18:36
🔗
|
w0rp |
I don't think I've ever seen a <form> before where one of the fields is marked as deprecated. |
|
18:51
🔗
|
w0rp |
https://archive.org/details/Wiki-Tanasinn.info Case probably needs fixing, but there it is. |
|
19:06
🔗
|
balrog |
Nemo_bis: what's the issue wrt wait times? |
|
19:06
🔗
|
balrog |
have you tried wait times with a random bias? |
|
19:07
🔗
|
Nemo_bis |
balrog: those sites are too stupid to do such a check :) they just have extremely low throttle, it's hard to navigate them even manually on your browser |
|
19:07
🔗
|
balrog |
ugh... |
|
19:08
🔗
|
Nemo_bis |
balrog: but they don't give clear errors nor say what speed limit you should follow, so it's hard to understand what's going on |
|
19:08
🔗
|
Nemo_bis |
sometimes they stop responding at all, sometimes they serve a 500 or 403, sometimes they serve a bogus text warning, sometimes they redirect to some stupid location |
|
19:09
🔗
|
Nemo_bis |
and sometimes they just are genuinely down or they just have some broken page ^^ |
|
21:21
🔗
|
odie5533 |
What do the Request packets look like? |
|
21:22
🔗
|
odie5533 |
if they are down or have a broken page, then that's nothing that can be fixed except to try back later |
|
21:23
🔗
|
Nemo_bis |
request packets? |
|
21:23
🔗
|
odie5533 |
HTTP Requests |
|
21:23
🔗
|
Nemo_bis |
and yes, that's why it's hard to identify what's going wrong and what to retry at what speed |
|
21:24
🔗
|
Nemo_bis |
odie5533: see yourself https://code.google.com/p/wikiteam/source/browse/trunk/dumpgenerator.py |
|
21:25
🔗
|
odie5533 |
I see... a lot of code. |
|
21:26
🔗
|
Nemo_bis |
odie5533: e.g. https://code.google.com/p/wikiteam/source/browse/trunk/dumpgenerator.py#286 |
|
21:28
🔗
|
odie5533 |
you are getting those "We have retried x times" errors? |
|
21:28
🔗
|
Nemo_bis |
sometimes |
|
21:29
🔗
|
odie5533 |
which error are you getting mostly, and on what url? |
|
21:29
🔗
|
Nemo_bis |
currently I'm not getting any error, I fixed them all |
|
21:30
🔗
|
Nemo_bis |
the errors the script gives are not particularly important, they're all lies ;) |
|
21:31
🔗
|
Nemo_bis |
the actual problems are 500, 403, 404, 307 and bogus 200 |
|
21:35
🔗
|
odie5533 |
let me know if you get it again, and on what url |
|
21:36
🔗
|
Nemo_bis |
it's not important, it's any URL; they're throttles.. |
|
21:39
🔗
|
odie5533 |
there's not much that can solve that except using more IPs, or slowing down the rate |
|
21:39
🔗
|
odie5533 |
might be good to have the program automatically the determine the rate |
|
21:41
🔗
|
Nemo_bis |
if only it knew how to determine if it was throttled :) |
|
21:42
🔗
|
Nemo_bis |
one which gives a different issue is dumpgenerator.py --index=http://ac.wikkii.com/w/index.php --xml --images : it fails with urllib complaining about 302 and redirect loops while downloading the list of titles |
|
21:53
🔗
|
odie5533 |
perhaps there is a redirect loop? |
|
21:53
🔗
|
odie5533 |
wiki won't even load for me. |
|
21:56
🔗
|
odie5533 |
there's no pages on that wiki |
|
22:05
🔗
|
Nemo_bis |
wrong |
|
22:30
🔗
|
Nemo_bis |
odie5533: a specific case in which one may succeed I filed here: https://code.google.com/p/wikiteam/issues/detail?id=70 |