#archiveteam-bs 2018-07-29,Sun

↑back Search

Time Nickname Message
00:16 🔗 Mateon1 has quit IRC (Remote host closed the connection)
00:16 🔗 Mateon1 has joined #archiveteam-bs
00:51 🔗 ta9le has quit IRC (Quit: Connection closed for inactivity)
01:07 🔗 godane SketchCow: i'm uploading another big file called 'Monty Python Flying Circus 6H CC WOC and VH1 1995-01-01.mpg'
01:08 🔗 godane file size is 17705459712
01:12 🔗 Flashfire I love monty python
01:18 🔗 SketchCow Great
01:24 🔗 Not_Avail has left
01:47 🔗 Mateon1 has quit IRC (Remote host closed the connection)
01:47 🔗 Mateon1 has joined #archiveteam-bs
01:52 🔗 godane has quit IRC (Read error: Operation timed out)
02:07 🔗 godane has joined #archiveteam-bs
02:08 🔗 Mateon1 has quit IRC (Remote host closed the connection)
02:09 🔗 Mateon1 has joined #archiveteam-bs
02:19 🔗 BlueMax has quit IRC (Read error: Connection reset by peer)
02:38 🔗 Mateon1 has quit IRC (Remote host closed the connection)
02:38 🔗 Mateon1 has joined #archiveteam-bs
02:44 🔗 Mateon1 has quit IRC (Remote host closed the connection)
02:45 🔗 Mateon1 has joined #archiveteam-bs
02:48 🔗 Mateon1 has quit IRC (Remote host closed the connection)
02:48 🔗 Mateon1 has joined #archiveteam-bs
03:23 🔗 chfoo-_ is now known as chfoo
03:33 🔗 archodg__ has joined #archiveteam-bs
03:35 🔗 odemg has quit IRC (Ping timeout: 268 seconds)
03:37 🔗 archodg_ has quit IRC (Read error: Operation timed out)
03:39 🔗 Arrhenius has joined #archiveteam-bs
03:47 🔗 odemg has joined #archiveteam-bs
04:02 🔗 espes__ has quit IRC (Ping timeout: 268 seconds)
04:07 🔗 Mateon1 has quit IRC (Remote host closed the connection)
04:09 🔗 Mateon1 has joined #archiveteam-bs
04:19 🔗 espes__ has joined #archiveteam-bs
05:13 🔗 zhongfu has quit IRC (Ping timeout: 260 seconds)
06:18 🔗 BlueMax has joined #archiveteam-bs
06:47 🔗 wp494 has quit IRC (Ping timeout: 268 seconds)
06:48 🔗 wp494 has joined #archiveteam-bs
07:09 🔗 zhongfu has joined #archiveteam-bs
09:58 🔗 ta9le has joined #archiveteam-bs
10:03 🔗 Dark_Star has quit IRC (Ping timeout: 246 seconds)
10:07 🔗 Darkstar has joined #archiveteam-bs
10:18 🔗 schbirid has joined #archiveteam-bs
10:36 🔗 BlueMax has quit IRC (Read error: Connection reset by peer)
11:22 🔗 DragonMon has joined #archiveteam-bs
12:45 🔗 jschwart has quit IRC (Quit: Konversation terminated!)
12:55 🔗 Dimtree has quit IRC (Read error: Connection reset by peer)
13:18 🔗 Dimtree has joined #archiveteam-bs
13:18 🔗 moufu_ has quit IRC (Read error: Operation timed out)
13:20 🔗 robogoat has quit IRC (Ping timeout: 252 seconds)
13:34 🔗 vegmitemo has joined #archiveteam-bs
13:37 🔗 vegmitemo Hi. Does anyone have any tips for handling 503 errors from archive.org? I've tried waiting hours between requests but it doesn't seem to make a difference.
13:41 🔗 BnAboyZ has joined #archiveteam-bs
13:43 🔗 robogoat has joined #archiveteam-bs
13:44 🔗 moufu has joined #archiveteam-bs
14:05 🔗 Mateon1 has quit IRC (Ping timeout: 255 seconds)
14:05 🔗 Mateon1 has joined #archiveteam-bs
14:07 🔗 SN4T14 has quit IRC (Remote host closed the connection)
14:07 🔗 SN4T14 has joined #archiveteam-bs
14:12 🔗 SN4T14 has quit IRC (Ping timeout: 268 seconds)
14:20 🔗 SN4T14 has joined #archiveteam-bs
14:39 🔗 Sk2d has joined #archiveteam-bs
14:41 🔗 Sk1d has quit IRC (Read error: Operation timed out)
14:41 🔗 Sk2d is now known as Sk1d
14:57 🔗 znak vegmitemo: may not apply to your situation, but I would get 503 errors when doing big uploads with s3cmd, because it would do the upload as many small requests. The workaround was to use the --disable-multipart option. Now I use the ia command-line client and I haven't had a problem.
15:00 🔗 vegmitemo znak, you mean for each file in the item? I'm only starting new connections for each item but all the files are part of that request. I guess if I kept reusing the established connection that might work. Any idea how long your blocked for with a 503?
15:13 🔗 Mateon1 has quit IRC (Remote host closed the connection)
15:17 🔗 Mateon1 has joined #archiveteam-bs
15:21 🔗 DragonMon has quit IRC (Ping timeout: 252 seconds)
15:32 🔗 Arrhenius has quit IRC ()
15:35 🔗 ozlo has quit IRC (Read error: Connection reset by peer)
15:35 🔗 ozlo_ has joined #archiveteam-bs
15:58 🔗 godane so i found out there is a magazine called Hero Illustrated
15:59 🔗 godane my brother has one of the magazines so i figure i would check if it got any scans online
15:59 🔗 godane turns out it does have scans online but not on archive.org
15:59 🔗 godane so i'm grabbing the scans found so you guys can have them
16:21 🔗 znak vegmitemo: s3cmd would break single files into 50 MB parts, I think. I don't know how long a 503 block lasts. It sounds like you're encountering something different than I was.
16:31 🔗 vegmitemo znak, it probably is. It seems weird the block would last this long, been a couple days since. Hopefully when it does finish reusing the connection will work. Kinda wish the errors from S3 were a bit more informational. Thanks.
16:53 🔗 TC01 has quit IRC (Read error: Operation timed out)
16:53 🔗 godane so i'm down to 16 tapes that maybe 4+ hours long
16:54 🔗 godane i still have a one other box of tapes but there all 2 hours or less
17:19 🔗 godane i'm uploading a file named 'top-100-of-80s-98-to-29-mtv-incomplete-part1-1989-12.mpg'
17:19 🔗 godane file size is 16,079,841,280
17:32 🔗 ta9le has quit IRC (Quit: Connection closed for inactivity)
18:45 🔗 SimpBrain has quit IRC (Read error: Operation timed out)
20:50 🔗 brotherof has joined #archiveteam-bs
20:51 🔗 brotherof hey
20:51 🔗 brotherof this is the account for my brother
20:51 🔗 brotherof i'm godane
20:55 🔗 jschwart has joined #archiveteam-bs
20:57 🔗 jschwart has quit IRC (Client Quit)
21:00 🔗 godane i'm back in my room
21:01 🔗 vegmitemo godane, any assistance? Thanks.
21:02 🔗 godane i have not seen the 503 errors today
21:02 🔗 brotherof has left
21:02 🔗 vegmitemo godane, I'm interested in what your solution was, in the logs you seemed to have figured out how to handle them.
21:03 🔗 godane i may have put the ip address into my hosts file
21:05 🔗 godane ping: http://s3.us.archive.org/: Temporary failure in name resolution
21:05 🔗 godane i'm getting that for a error when ping it
21:06 🔗 vegmitemo godane, I'm not sure it's a general S3 issue, seems like my keys in particular are getting the 503, I'd only managed about 13 items before then so it's very strange.
21:06 🔗 JAA Resolves fine for me, but pinging fails with a "Destination Port Unreachable" error.
21:07 🔗 vegmitemo What makes even less sense is I've been leaving hours, a day or so even between requests. Very odd.
21:09 🔗 vegmitemo godane, my bad, it was riking who had figured out something.
21:09 🔗 schbirid has quit IRC (Quit: Leaving)
21:09 🔗 godane 4 packets transmitted, 0 received, +4 errors, 100% packet loss, time 3003ms
21:10 🔗 godane JAA: what is s3.us.archive.org ip address?
21:11 🔗 JAA godane: 4 A records: 207.241.225.84 207.241.225.184 207.241.225.48 207.241.225.51
21:11 🔗 JAA No AAAA record. IA, do you even IPv6?
21:12 🔗 vegmitemo What was that parameter you could add to s3.us.archive.org in a browser to see how many tasks there were, that sort of thing? Last I checked it there was no ratelimitting or anything, not even on my access key.
21:12 🔗 vegmitemo Even then though I couldn't upload.
21:14 🔗 ta9le has joined #archiveteam-bs
21:15 🔗 JAA vegmitemo: Search for 503 on https://archive.org/help/abouts3.txt
21:16 🔗 vegmitemo JAA, got it, it's ?check_limit=1.
21:16 🔗 riking Hello!
21:17 🔗 riking Please repeat what the question was?
21:17 🔗 vegmitemo riking, hey. I was wondering about how you handled 503 errors. I haven't been able to upload for a couple days, even with hours between attempts.
21:18 🔗 riking Sleep for Retry-After, if present, and Reinsert the upload into my job queue
21:19 🔗 vegmitemo riking, a header containing a wait period?
21:21 🔗 riking Line "if resp.Status == "503 Slow Down" {" here https://github.com/riking/mspfa-archiver/blob/master/upload.go
21:21 🔗 riking Oh actually it's just a one minute sleep
21:22 🔗 vegmitemo Just tried a request, didn't seem to get a Retry-After header back.
21:23 🔗 riking Yeah there's no retry-after, it's a special status message
21:24 🔗 riking StatusCode == 503 and StatusMessage == "Slow Down"
21:24 🔗 riking That means your account has too many jobs queued
21:25 🔗 vegmitemo riking, when I plugged my access key into https://s3.us.archive.org/?check_limit=1 I don't actually have any jobs going. My items have already derived too.
21:25 🔗 riking Above on line "Resubmit failing jobs" is the actual resumbit, which happens on any error
21:26 🔗 riking If it's a non-Slow Down 503, you just need to wait for the situation to resolve. Maybe you got unlucky on your dns randomization
21:26 🔗 vegmitemo riking, I'm starting to think that too. My script was already sleeping when a 503 was recieved and even after an hour sleep it still wasn't happy.
21:27 🔗 riking Yeah no retry fast if it's not a slow down
21:28 🔗 vegmitemo riking, tried that too, no dice.
21:28 🔗 riking Hmm does python have internal dns caching
21:29 🔗 riking I know that go is a bit crazy and reimplementated a dns resolver which supports round robin results
21:29 🔗 vegmitemo riking, I'll be back in a bit. Using socks5h so shouldn't be?
21:30 🔗 riking So if we get 4 IPs back and one is bad, 1/4 requests will retry for a good server
21:31 🔗 riking See if you can disable dns caching somewhere, that might let you reroll the dice
22:47 🔗 godane vegmitemo: i was able to upload to DTIC docs today
22:47 🔗 godane using S3
22:58 🔗 SimpBrain has joined #archiveteam-bs
23:08 🔗 godane i'm digitizing bilbschirm 25
23:20 🔗 vegmitemo has quit IRC (Remote host closed the connection)
23:43 🔗 vegmitemo has joined #archiveteam-bs
23:44 🔗 vegmitemo riking, flushed my cache a number of times, S3 doesn't seem to like me. Not sure what I've done to piss it off.
23:44 🔗 riking Are you getting anything at all uploaded?
23:45 🔗 riking Always possible there's a service outage
23:45 🔗 riking Which is the original meaning of 503
23:46 🔗 vegmitemo Here's my page: https://archive.org/details/@vegmitemong after that last one it just keeps 503'ing.
23:46 🔗 riking Do you have your code up somewhere
23:48 🔗 vegmitemo Not yet, need to clean up a bit first. It's nothing amazing, just pycurl automated really.
23:49 🔗 vegmitemo It hasn't been an issue until now, so not sure what I could change to appease S3.
23:50 🔗 riking how do i filter to someone else's tasks uhh
23:53 🔗 vegmitemo "accesskey_tasks_queued": 0 from s3.us.archive.org/?check_limit=1&accesskey=<myaccesskey>
23:53 🔗 vegmitemo "bucket_tasks_queued": 0 also.
23:54 🔗 vegmitemo Unless there's some new user limit or something?
23:54 🔗 riking you've got a bunch of dead derives
23:54 🔗 vegmitemo Which item?
23:54 🔗 riking https://usercontent.irccloud-cdn.com/file/edbcoxm2/image.png
23:55 🔗 riking should i try rerunning the red ones?
23:56 🔗 vegmitemo I've not uploaded any anime or manga, how can that be on my account? Weird.
23:56 🔗 riking oh is sketch@cow not yours
23:56 🔗 riking i saw they had a taskon one of your items
23:56 🔗 riking (screensoht deleted)
23:57 🔗 vegmitemo That's Jason Scott isn't it?
23:57 🔗 riking oh it probably is lol
23:57 🔗 vegmitemo Oh I know why he's got one.
23:57 🔗 riking ah, change collection
23:57 🔗 vegmitemo Stuff with the youtube- prefix is moved into the mirrortube and archiveteam collections.
23:58 🔗 riking anyways yeah, "b73ejjkp4t@el6vq.anonbox.net" doesn't have any jobs for the last 4 days.
23:59 🔗 vegmitemo What is this like a shadowban or something? Hah.
23:59 🔗 riking did you change your upload script on the 25th?

irclogger-viewer