[00:16] *** Mateon1 has quit IRC (Remote host closed the connection) [00:16] *** Mateon1 has joined #archiveteam-bs [00:51] *** ta9le has quit IRC (Quit: Connection closed for inactivity) [01:07] SketchCow: i'm uploading another big file called 'Monty Python Flying Circus 6H CC WOC and VH1 1995-01-01.mpg' [01:08] file size is 17705459712 [01:12] I love monty python [01:18] Great [01:24] *** Not_Avail has left [01:47] *** Mateon1 has quit IRC (Remote host closed the connection) [01:47] *** Mateon1 has joined #archiveteam-bs [01:52] *** godane has quit IRC (Read error: Operation timed out) [02:07] *** godane has joined #archiveteam-bs [02:08] *** Mateon1 has quit IRC (Remote host closed the connection) [02:09] *** Mateon1 has joined #archiveteam-bs [02:19] *** BlueMax has quit IRC (Read error: Connection reset by peer) [02:38] *** Mateon1 has quit IRC (Remote host closed the connection) [02:38] *** Mateon1 has joined #archiveteam-bs [02:44] *** Mateon1 has quit IRC (Remote host closed the connection) [02:45] *** Mateon1 has joined #archiveteam-bs [02:48] *** Mateon1 has quit IRC (Remote host closed the connection) [02:48] *** Mateon1 has joined #archiveteam-bs [03:23] *** chfoo-_ is now known as chfoo [03:33] *** archodg__ has joined #archiveteam-bs [03:35] *** odemg has quit IRC (Ping timeout: 268 seconds) [03:37] *** archodg_ has quit IRC (Read error: Operation timed out) [03:39] *** Arrhenius has joined #archiveteam-bs [03:47] *** odemg has joined #archiveteam-bs [04:02] *** espes__ has quit IRC (Ping timeout: 268 seconds) [04:07] *** Mateon1 has quit IRC (Remote host closed the connection) [04:09] *** Mateon1 has joined #archiveteam-bs [04:19] *** espes__ has joined #archiveteam-bs [05:13] *** zhongfu has quit IRC (Ping timeout: 260 seconds) [06:18] *** BlueMax has joined #archiveteam-bs [06:47] *** wp494 has quit IRC (Ping timeout: 268 seconds) [06:48] *** wp494 has joined #archiveteam-bs [07:09] *** zhongfu has joined #archiveteam-bs [09:58] *** ta9le has joined #archiveteam-bs [10:03] *** Dark_Star has quit IRC (Ping timeout: 246 seconds) [10:07] *** Darkstar has joined #archiveteam-bs [10:18] *** schbirid has joined #archiveteam-bs [10:36] *** BlueMax has quit IRC (Read error: Connection reset by peer) [11:22] *** DragonMon has joined #archiveteam-bs [12:45] *** jschwart has quit IRC (Quit: Konversation terminated!) [12:55] *** Dimtree has quit IRC (Read error: Connection reset by peer) [13:18] *** Dimtree has joined #archiveteam-bs [13:18] *** moufu_ has quit IRC (Read error: Operation timed out) [13:20] *** robogoat has quit IRC (Ping timeout: 252 seconds) [13:34] *** vegmitemo has joined #archiveteam-bs [13:37] Hi. Does anyone have any tips for handling 503 errors from archive.org? I've tried waiting hours between requests but it doesn't seem to make a difference. [13:41] *** BnAboyZ has joined #archiveteam-bs [13:43] *** robogoat has joined #archiveteam-bs [13:44] *** moufu has joined #archiveteam-bs [14:05] *** Mateon1 has quit IRC (Ping timeout: 255 seconds) [14:05] *** Mateon1 has joined #archiveteam-bs [14:07] *** SN4T14 has quit IRC (Remote host closed the connection) [14:07] *** SN4T14 has joined #archiveteam-bs [14:12] *** SN4T14 has quit IRC (Ping timeout: 268 seconds) [14:20] *** SN4T14 has joined #archiveteam-bs [14:39] *** Sk2d has joined #archiveteam-bs [14:41] *** Sk1d has quit IRC (Read error: Operation timed out) [14:41] *** Sk2d is now known as Sk1d [14:57] vegmitemo: may not apply to your situation, but I would get 503 errors when doing big uploads with s3cmd, because it would do the upload as many small requests. The workaround was to use the --disable-multipart option. Now I use the ia command-line client and I haven't had a problem. [15:00] znak, you mean for each file in the item? I'm only starting new connections for each item but all the files are part of that request. I guess if I kept reusing the established connection that might work. Any idea how long your blocked for with a 503? [15:13] *** Mateon1 has quit IRC (Remote host closed the connection) [15:17] *** Mateon1 has joined #archiveteam-bs [15:21] *** DragonMon has quit IRC (Ping timeout: 252 seconds) [15:32] *** Arrhenius has quit IRC () [15:35] *** ozlo has quit IRC (Read error: Connection reset by peer) [15:35] *** ozlo_ has joined #archiveteam-bs [15:58] so i found out there is a magazine called Hero Illustrated [15:59] my brother has one of the magazines so i figure i would check if it got any scans online [15:59] turns out it does have scans online but not on archive.org [15:59] so i'm grabbing the scans found so you guys can have them [16:21] vegmitemo: s3cmd would break single files into 50 MB parts, I think. I don't know how long a 503 block lasts. It sounds like you're encountering something different than I was. [16:31] znak, it probably is. It seems weird the block would last this long, been a couple days since. Hopefully when it does finish reusing the connection will work. Kinda wish the errors from S3 were a bit more informational. Thanks. [16:53] *** TC01 has quit IRC (Read error: Operation timed out) [16:53] so i'm down to 16 tapes that maybe 4+ hours long [16:54] i still have a one other box of tapes but there all 2 hours or less [17:19] i'm uploading a file named 'top-100-of-80s-98-to-29-mtv-incomplete-part1-1989-12.mpg' [17:19] file size is 16,079,841,280 [17:32] *** ta9le has quit IRC (Quit: Connection closed for inactivity) [18:45] *** SimpBrain has quit IRC (Read error: Operation timed out) [20:50] *** brotherof has joined #archiveteam-bs [20:51] hey [20:51] this is the account for my brother [20:51] i'm godane [20:55] *** jschwart has joined #archiveteam-bs [20:57] *** jschwart has quit IRC (Client Quit) [21:00] i'm back in my room [21:01] godane, any assistance? Thanks. [21:02] i have not seen the 503 errors today [21:02] *** brotherof has left [21:02] godane, I'm interested in what your solution was, in the logs you seemed to have figured out how to handle them. [21:03] i may have put the ip address into my hosts file [21:05] ping: http://s3.us.archive.org/: Temporary failure in name resolution [21:05] i'm getting that for a error when ping it [21:06] godane, I'm not sure it's a general S3 issue, seems like my keys in particular are getting the 503, I'd only managed about 13 items before then so it's very strange. [21:06] Resolves fine for me, but pinging fails with a "Destination Port Unreachable" error. [21:07] What makes even less sense is I've been leaving hours, a day or so even between requests. Very odd. [21:09] godane, my bad, it was riking who had figured out something. [21:09] *** schbirid has quit IRC (Quit: Leaving) [21:09] 4 packets transmitted, 0 received, +4 errors, 100% packet loss, time 3003ms [21:10] JAA: what is s3.us.archive.org ip address? [21:11] godane: 4 A records: 207.241.225.84 207.241.225.184 207.241.225.48 207.241.225.51 [21:11] No AAAA record. IA, do you even IPv6? [21:12] What was that parameter you could add to s3.us.archive.org in a browser to see how many tasks there were, that sort of thing? Last I checked it there was no ratelimitting or anything, not even on my access key. [21:12] Even then though I couldn't upload. [21:14] *** ta9le has joined #archiveteam-bs [21:15] vegmitemo: Search for 503 on https://archive.org/help/abouts3.txt [21:16] JAA, got it, it's ?check_limit=1. [21:16] Hello! [21:17] Please repeat what the question was? [21:17] riking, hey. I was wondering about how you handled 503 errors. I haven't been able to upload for a couple days, even with hours between attempts. [21:18] Sleep for Retry-After, if present, and Reinsert the upload into my job queue [21:19] riking, a header containing a wait period? [21:21] Line "if resp.Status == "503 Slow Down" {" here https://github.com/riking/mspfa-archiver/blob/master/upload.go [21:21] Oh actually it's just a one minute sleep [21:22] Just tried a request, didn't seem to get a Retry-After header back. [21:23] Yeah there's no retry-after, it's a special status message [21:24] StatusCode == 503 and StatusMessage == "Slow Down" [21:24] That means your account has too many jobs queued [21:25] riking, when I plugged my access key into https://s3.us.archive.org/?check_limit=1 I don't actually have any jobs going. My items have already derived too. [21:25] Above on line "Resubmit failing jobs" is the actual resumbit, which happens on any error [21:26] If it's a non-Slow Down 503, you just need to wait for the situation to resolve. Maybe you got unlucky on your dns randomization [21:26] riking, I'm starting to think that too. My script was already sleeping when a 503 was recieved and even after an hour sleep it still wasn't happy. [21:27] Yeah no retry fast if it's not a slow down [21:28] riking, tried that too, no dice. [21:28] Hmm does python have internal dns caching [21:29] I know that go is a bit crazy and reimplementated a dns resolver which supports round robin results [21:29] riking, I'll be back in a bit. Using socks5h so shouldn't be? [21:30] So if we get 4 IPs back and one is bad, 1/4 requests will retry for a good server [21:31] See if you can disable dns caching somewhere, that might let you reroll the dice [22:47] vegmitemo: i was able to upload to DTIC docs today [22:47] using S3 [22:58] *** SimpBrain has joined #archiveteam-bs [23:08] i'm digitizing bilbschirm 25 [23:20] *** vegmitemo has quit IRC (Remote host closed the connection) [23:43] *** vegmitemo has joined #archiveteam-bs [23:44] riking, flushed my cache a number of times, S3 doesn't seem to like me. Not sure what I've done to piss it off. [23:44] Are you getting anything at all uploaded? [23:45] Always possible there's a service outage [23:45] Which is the original meaning of 503 [23:46] Here's my page: https://archive.org/details/@vegmitemong after that last one it just keeps 503'ing. [23:46] Do you have your code up somewhere [23:48] Not yet, need to clean up a bit first. It's nothing amazing, just pycurl automated really. [23:49] It hasn't been an issue until now, so not sure what I could change to appease S3. [23:50] how do i filter to someone else's tasks uhh [23:53] "accesskey_tasks_queued": 0 from s3.us.archive.org/?check_limit=1&accesskey= [23:53] "bucket_tasks_queued": 0 also. [23:54] Unless there's some new user limit or something? [23:54] you've got a bunch of dead derives [23:54] Which item? [23:54] https://usercontent.irccloud-cdn.com/file/edbcoxm2/image.png [23:55] should i try rerunning the red ones? [23:56] I've not uploaded any anime or manga, how can that be on my account? Weird. [23:56] oh is sketch@cow not yours [23:56] i saw they had a taskon one of your items [23:56] (screensoht deleted) [23:57] That's Jason Scott isn't it? [23:57] oh it probably is lol [23:57] Oh I know why he's got one. [23:57] ah, change collection [23:57] Stuff with the youtube- prefix is moved into the mirrortube and archiveteam collections. [23:58] anyways yeah, "b73ejjkp4t@el6vq.anonbox.net" doesn't have any jobs for the last 4 days. [23:59] What is this like a shadowban or something? Hah. [23:59] did you change your upload script on the 25th?