| Time |
Nickname |
Message |
|
00:16
🔗
|
|
Mateon1 has quit IRC (Remote host closed the connection) |
|
00:16
🔗
|
|
Mateon1 has joined #archiveteam-bs |
|
00:51
🔗
|
|
ta9le has quit IRC (Quit: Connection closed for inactivity) |
|
01:07
🔗
|
godane |
SketchCow: i'm uploading another big file called 'Monty Python Flying Circus 6H CC WOC and VH1 1995-01-01.mpg' |
|
01:08
🔗
|
godane |
file size is 17705459712 |
|
01:12
🔗
|
Flashfire |
I love monty python |
|
01:18
🔗
|
SketchCow |
Great |
|
01:24
🔗
|
|
Not_Avail has left |
|
01:47
🔗
|
|
Mateon1 has quit IRC (Remote host closed the connection) |
|
01:47
🔗
|
|
Mateon1 has joined #archiveteam-bs |
|
01:52
🔗
|
|
godane has quit IRC (Read error: Operation timed out) |
|
02:07
🔗
|
|
godane has joined #archiveteam-bs |
|
02:08
🔗
|
|
Mateon1 has quit IRC (Remote host closed the connection) |
|
02:09
🔗
|
|
Mateon1 has joined #archiveteam-bs |
|
02:19
🔗
|
|
BlueMax has quit IRC (Read error: Connection reset by peer) |
|
02:38
🔗
|
|
Mateon1 has quit IRC (Remote host closed the connection) |
|
02:38
🔗
|
|
Mateon1 has joined #archiveteam-bs |
|
02:44
🔗
|
|
Mateon1 has quit IRC (Remote host closed the connection) |
|
02:45
🔗
|
|
Mateon1 has joined #archiveteam-bs |
|
02:48
🔗
|
|
Mateon1 has quit IRC (Remote host closed the connection) |
|
02:48
🔗
|
|
Mateon1 has joined #archiveteam-bs |
|
03:23
🔗
|
|
chfoo-_ is now known as chfoo |
|
03:33
🔗
|
|
archodg__ has joined #archiveteam-bs |
|
03:35
🔗
|
|
odemg has quit IRC (Ping timeout: 268 seconds) |
|
03:37
🔗
|
|
archodg_ has quit IRC (Read error: Operation timed out) |
|
03:39
🔗
|
|
Arrhenius has joined #archiveteam-bs |
|
03:47
🔗
|
|
odemg has joined #archiveteam-bs |
|
04:02
🔗
|
|
espes__ has quit IRC (Ping timeout: 268 seconds) |
|
04:07
🔗
|
|
Mateon1 has quit IRC (Remote host closed the connection) |
|
04:09
🔗
|
|
Mateon1 has joined #archiveteam-bs |
|
04:19
🔗
|
|
espes__ has joined #archiveteam-bs |
|
05:13
🔗
|
|
zhongfu has quit IRC (Ping timeout: 260 seconds) |
|
06:18
🔗
|
|
BlueMax has joined #archiveteam-bs |
|
06:47
🔗
|
|
wp494 has quit IRC (Ping timeout: 268 seconds) |
|
06:48
🔗
|
|
wp494 has joined #archiveteam-bs |
|
07:09
🔗
|
|
zhongfu has joined #archiveteam-bs |
|
09:58
🔗
|
|
ta9le has joined #archiveteam-bs |
|
10:03
🔗
|
|
Dark_Star has quit IRC (Ping timeout: 246 seconds) |
|
10:07
🔗
|
|
Darkstar has joined #archiveteam-bs |
|
10:18
🔗
|
|
schbirid has joined #archiveteam-bs |
|
10:36
🔗
|
|
BlueMax has quit IRC (Read error: Connection reset by peer) |
|
11:22
🔗
|
|
DragonMon has joined #archiveteam-bs |
|
12:45
🔗
|
|
jschwart has quit IRC (Quit: Konversation terminated!) |
|
12:55
🔗
|
|
Dimtree has quit IRC (Read error: Connection reset by peer) |
|
13:18
🔗
|
|
Dimtree has joined #archiveteam-bs |
|
13:18
🔗
|
|
moufu_ has quit IRC (Read error: Operation timed out) |
|
13:20
🔗
|
|
robogoat has quit IRC (Ping timeout: 252 seconds) |
|
13:34
🔗
|
|
vegmitemo has joined #archiveteam-bs |
|
13:37
🔗
|
vegmitemo |
Hi. Does anyone have any tips for handling 503 errors from archive.org? I've tried waiting hours between requests but it doesn't seem to make a difference. |
|
13:41
🔗
|
|
BnAboyZ has joined #archiveteam-bs |
|
13:43
🔗
|
|
robogoat has joined #archiveteam-bs |
|
13:44
🔗
|
|
moufu has joined #archiveteam-bs |
|
14:05
🔗
|
|
Mateon1 has quit IRC (Ping timeout: 255 seconds) |
|
14:05
🔗
|
|
Mateon1 has joined #archiveteam-bs |
|
14:07
🔗
|
|
SN4T14 has quit IRC (Remote host closed the connection) |
|
14:07
🔗
|
|
SN4T14 has joined #archiveteam-bs |
|
14:12
🔗
|
|
SN4T14 has quit IRC (Ping timeout: 268 seconds) |
|
14:20
🔗
|
|
SN4T14 has joined #archiveteam-bs |
|
14:39
🔗
|
|
Sk2d has joined #archiveteam-bs |
|
14:41
🔗
|
|
Sk1d has quit IRC (Read error: Operation timed out) |
|
14:41
🔗
|
|
Sk2d is now known as Sk1d |
|
14:57
🔗
|
znak |
vegmitemo: may not apply to your situation, but I would get 503 errors when doing big uploads with s3cmd, because it would do the upload as many small requests. The workaround was to use the --disable-multipart option. Now I use the ia command-line client and I haven't had a problem. |
|
15:00
🔗
|
vegmitemo |
znak, you mean for each file in the item? I'm only starting new connections for each item but all the files are part of that request. I guess if I kept reusing the established connection that might work. Any idea how long your blocked for with a 503? |
|
15:13
🔗
|
|
Mateon1 has quit IRC (Remote host closed the connection) |
|
15:17
🔗
|
|
Mateon1 has joined #archiveteam-bs |
|
15:21
🔗
|
|
DragonMon has quit IRC (Ping timeout: 252 seconds) |
|
15:32
🔗
|
|
Arrhenius has quit IRC () |
|
15:35
🔗
|
|
ozlo has quit IRC (Read error: Connection reset by peer) |
|
15:35
🔗
|
|
ozlo_ has joined #archiveteam-bs |
|
15:58
🔗
|
godane |
so i found out there is a magazine called Hero Illustrated |
|
15:59
🔗
|
godane |
my brother has one of the magazines so i figure i would check if it got any scans online |
|
15:59
🔗
|
godane |
turns out it does have scans online but not on archive.org |
|
15:59
🔗
|
godane |
so i'm grabbing the scans found so you guys can have them |
|
16:21
🔗
|
znak |
vegmitemo: s3cmd would break single files into 50 MB parts, I think. I don't know how long a 503 block lasts. It sounds like you're encountering something different than I was. |
|
16:31
🔗
|
vegmitemo |
znak, it probably is. It seems weird the block would last this long, been a couple days since. Hopefully when it does finish reusing the connection will work. Kinda wish the errors from S3 were a bit more informational. Thanks. |
|
16:53
🔗
|
|
TC01 has quit IRC (Read error: Operation timed out) |
|
16:53
🔗
|
godane |
so i'm down to 16 tapes that maybe 4+ hours long |
|
16:54
🔗
|
godane |
i still have a one other box of tapes but there all 2 hours or less |
|
17:19
🔗
|
godane |
i'm uploading a file named 'top-100-of-80s-98-to-29-mtv-incomplete-part1-1989-12.mpg' |
|
17:19
🔗
|
godane |
file size is 16,079,841,280 |
|
17:32
🔗
|
|
ta9le has quit IRC (Quit: Connection closed for inactivity) |
|
18:45
🔗
|
|
SimpBrain has quit IRC (Read error: Operation timed out) |
|
20:50
🔗
|
|
brotherof has joined #archiveteam-bs |
|
20:51
🔗
|
brotherof |
hey |
|
20:51
🔗
|
brotherof |
this is the account for my brother |
|
20:51
🔗
|
brotherof |
i'm godane |
|
20:55
🔗
|
|
jschwart has joined #archiveteam-bs |
|
20:57
🔗
|
|
jschwart has quit IRC (Client Quit) |
|
21:00
🔗
|
godane |
i'm back in my room |
|
21:01
🔗
|
vegmitemo |
godane, any assistance? Thanks. |
|
21:02
🔗
|
godane |
i have not seen the 503 errors today |
|
21:02
🔗
|
|
brotherof has left |
|
21:02
🔗
|
vegmitemo |
godane, I'm interested in what your solution was, in the logs you seemed to have figured out how to handle them. |
|
21:03
🔗
|
godane |
i may have put the ip address into my hosts file |
|
21:05
🔗
|
godane |
ping: http://s3.us.archive.org/: Temporary failure in name resolution |
|
21:05
🔗
|
godane |
i'm getting that for a error when ping it |
|
21:06
🔗
|
vegmitemo |
godane, I'm not sure it's a general S3 issue, seems like my keys in particular are getting the 503, I'd only managed about 13 items before then so it's very strange. |
|
21:06
🔗
|
JAA |
Resolves fine for me, but pinging fails with a "Destination Port Unreachable" error. |
|
21:07
🔗
|
vegmitemo |
What makes even less sense is I've been leaving hours, a day or so even between requests. Very odd. |
|
21:09
🔗
|
vegmitemo |
godane, my bad, it was riking who had figured out something. |
|
21:09
🔗
|
|
schbirid has quit IRC (Quit: Leaving) |
|
21:09
🔗
|
godane |
4 packets transmitted, 0 received, +4 errors, 100% packet loss, time 3003ms |
|
21:10
🔗
|
godane |
JAA: what is s3.us.archive.org ip address? |
|
21:11
🔗
|
JAA |
godane: 4 A records: 207.241.225.84 207.241.225.184 207.241.225.48 207.241.225.51 |
|
21:11
🔗
|
JAA |
No AAAA record. IA, do you even IPv6? |
|
21:12
🔗
|
vegmitemo |
What was that parameter you could add to s3.us.archive.org in a browser to see how many tasks there were, that sort of thing? Last I checked it there was no ratelimitting or anything, not even on my access key. |
|
21:12
🔗
|
vegmitemo |
Even then though I couldn't upload. |
|
21:14
🔗
|
|
ta9le has joined #archiveteam-bs |
|
21:15
🔗
|
JAA |
vegmitemo: Search for 503 on https://archive.org/help/abouts3.txt |
|
21:16
🔗
|
vegmitemo |
JAA, got it, it's ?check_limit=1. |
|
21:16
🔗
|
riking |
Hello! |
|
21:17
🔗
|
riking |
Please repeat what the question was? |
|
21:17
🔗
|
vegmitemo |
riking, hey. I was wondering about how you handled 503 errors. I haven't been able to upload for a couple days, even with hours between attempts. |
|
21:18
🔗
|
riking |
Sleep for Retry-After, if present, and Reinsert the upload into my job queue |
|
21:19
🔗
|
vegmitemo |
riking, a header containing a wait period? |
|
21:21
🔗
|
riking |
Line "if resp.Status == "503 Slow Down" {" here https://github.com/riking/mspfa-archiver/blob/master/upload.go |
|
21:21
🔗
|
riking |
Oh actually it's just a one minute sleep |
|
21:22
🔗
|
vegmitemo |
Just tried a request, didn't seem to get a Retry-After header back. |
|
21:23
🔗
|
riking |
Yeah there's no retry-after, it's a special status message |
|
21:24
🔗
|
riking |
StatusCode == 503 and StatusMessage == "Slow Down" |
|
21:24
🔗
|
riking |
That means your account has too many jobs queued |
|
21:25
🔗
|
vegmitemo |
riking, when I plugged my access key into https://s3.us.archive.org/?check_limit=1 I don't actually have any jobs going. My items have already derived too. |
|
21:25
🔗
|
riking |
Above on line "Resubmit failing jobs" is the actual resumbit, which happens on any error |
|
21:26
🔗
|
riking |
If it's a non-Slow Down 503, you just need to wait for the situation to resolve. Maybe you got unlucky on your dns randomization |
|
21:26
🔗
|
vegmitemo |
riking, I'm starting to think that too. My script was already sleeping when a 503 was recieved and even after an hour sleep it still wasn't happy. |
|
21:27
🔗
|
riking |
Yeah no retry fast if it's not a slow down |
|
21:28
🔗
|
vegmitemo |
riking, tried that too, no dice. |
|
21:28
🔗
|
riking |
Hmm does python have internal dns caching |
|
21:29
🔗
|
riking |
I know that go is a bit crazy and reimplementated a dns resolver which supports round robin results |
|
21:29
🔗
|
vegmitemo |
riking, I'll be back in a bit. Using socks5h so shouldn't be? |
|
21:30
🔗
|
riking |
So if we get 4 IPs back and one is bad, 1/4 requests will retry for a good server |
|
21:31
🔗
|
riking |
See if you can disable dns caching somewhere, that might let you reroll the dice |
|
22:47
🔗
|
godane |
vegmitemo: i was able to upload to DTIC docs today |
|
22:47
🔗
|
godane |
using S3 |
|
22:58
🔗
|
|
SimpBrain has joined #archiveteam-bs |
|
23:08
🔗
|
godane |
i'm digitizing bilbschirm 25 |
|
23:20
🔗
|
|
vegmitemo has quit IRC (Remote host closed the connection) |
|
23:43
🔗
|
|
vegmitemo has joined #archiveteam-bs |
|
23:44
🔗
|
vegmitemo |
riking, flushed my cache a number of times, S3 doesn't seem to like me. Not sure what I've done to piss it off. |
|
23:44
🔗
|
riking |
Are you getting anything at all uploaded? |
|
23:45
🔗
|
riking |
Always possible there's a service outage |
|
23:45
🔗
|
riking |
Which is the original meaning of 503 |
|
23:46
🔗
|
vegmitemo |
Here's my page: https://archive.org/details/@vegmitemong after that last one it just keeps 503'ing. |
|
23:46
🔗
|
riking |
Do you have your code up somewhere |
|
23:48
🔗
|
vegmitemo |
Not yet, need to clean up a bit first. It's nothing amazing, just pycurl automated really. |
|
23:49
🔗
|
vegmitemo |
It hasn't been an issue until now, so not sure what I could change to appease S3. |
|
23:50
🔗
|
riking |
how do i filter to someone else's tasks uhh |
|
23:53
🔗
|
vegmitemo |
"accesskey_tasks_queued": 0 from s3.us.archive.org/?check_limit=1&accesskey=<myaccesskey> |
|
23:53
🔗
|
vegmitemo |
"bucket_tasks_queued": 0 also. |
|
23:54
🔗
|
vegmitemo |
Unless there's some new user limit or something? |
|
23:54
🔗
|
riking |
you've got a bunch of dead derives |
|
23:54
🔗
|
vegmitemo |
Which item? |
|
23:54
🔗
|
riking |
https://usercontent.irccloud-cdn.com/file/edbcoxm2/image.png |
|
23:55
🔗
|
riking |
should i try rerunning the red ones? |
|
23:56
🔗
|
vegmitemo |
I've not uploaded any anime or manga, how can that be on my account? Weird. |
|
23:56
🔗
|
riking |
oh is sketch@cow not yours |
|
23:56
🔗
|
riking |
i saw they had a taskon one of your items |
|
23:56
🔗
|
riking |
(screensoht deleted) |
|
23:57
🔗
|
vegmitemo |
That's Jason Scott isn't it? |
|
23:57
🔗
|
riking |
oh it probably is lol |
|
23:57
🔗
|
vegmitemo |
Oh I know why he's got one. |
|
23:57
🔗
|
riking |
ah, change collection |
|
23:57
🔗
|
vegmitemo |
Stuff with the youtube- prefix is moved into the mirrortube and archiveteam collections. |
|
23:58
🔗
|
riking |
anyways yeah, "b73ejjkp4t@el6vq.anonbox.net" doesn't have any jobs for the last 4 days. |
|
23:59
🔗
|
vegmitemo |
What is this like a shadowban or something? Hah. |
|
23:59
🔗
|
riking |
did you change your upload script on the 25th? |