#urlteam 2019-04-19,Fri

↑back Search

Time Nickname Message
00:08 πŸ”— Flashfire I think the myspace urls might be done for the moment we got some 500s
00:09 πŸ”— Flashfire maybe not done but a lot of errors
00:11 πŸ”— Flashfire JAA you have any idea what that error means?
00:11 πŸ”— JAA Flashfire: Have an example URL?
00:12 πŸ”— Flashfire There are the errors in the error reports on the tracker
00:13 πŸ”— Flashfire I dont know what they mean
00:13 πŸ”— JAA Yeah, but I don't have my tracker password handy.
00:14 πŸ”— Flashfire I can change the password for you just DM me what you want it to be
00:14 πŸ”— JAA Nah
00:15 πŸ”— JAA Then I have to remember to update it in my password manager afterwards etc.
00:15 πŸ”— JAA I can take a look in a bit.
00:21 πŸ”— Flashfire ok
00:33 πŸ”— Flashfire http://ur.ly/
00:35 πŸ”— Flashfire https://shorteners.net/
00:37 πŸ”— JAA Flashfire: I don't see any 500 errors on mysp-ac, only timeouts.
00:37 πŸ”— Flashfire ok
00:37 πŸ”— JAA And we are still finding results.
00:39 πŸ”— JAA Oh wait, I was looking at the wrong page.
00:39 πŸ”— JAA Yeah, not many results recently for mysp-ac.
00:40 πŸ”— JAA Not sure what that means though.
00:40 πŸ”— Flashfire https://gram.im/h2PLR1gg
00:40 πŸ”— Flashfire New one I found
00:41 πŸ”— Flashfire Captchas yuck
00:41 πŸ”— JAA And 8-char codes are too long to bruteforce anyway.
00:41 πŸ”— Flashfire http://tmearn.com/zIMfR
00:42 πŸ”— Flashfire Connection reset for me for the url above
00:43 πŸ”— Flashfire JAA what do you get?
00:43 πŸ”— Flashfire http://oke.io/B5DqMWy
00:44 πŸ”— JAA Flashfire: For tmearn.com? Works fine here.
00:44 πŸ”— Flashfire Ok
00:44 πŸ”— Flashfire https://lyonkim.net/T7vHyb
00:46 πŸ”— Flashfire http://zlshorte.net/NkSKYuQI
00:46 πŸ”— Flashfire https://adpop.me/yEaDc0pl
01:07 πŸ”— Flashfire JAA I am not doing to bad with puri.na I had a hunch and a bit of proof and rolled with it so far found 40 urls that are different to the standard bitly ones
01:37 πŸ”— Zerote has quit IRC (Ping timeout: 260 seconds)
01:44 πŸ”— Jusque has quit IRC (Ping timeout: 600 seconds)
02:12 πŸ”— Jusque has joined #urlteam
02:46 πŸ”— ivan has quit IRC (Write error: Broken pipe)
02:47 πŸ”— kiska1 has quit IRC (Read error: Operation timed out)
02:47 πŸ”— Mayonaise has quit IRC (Read error: Operation timed out)
02:47 πŸ”— ivan has joined #urlteam
02:48 πŸ”— TigerbotH has quit IRC (Read error: Operation timed out)
02:48 πŸ”— Mayonaise has joined #urlteam
03:36 πŸ”— odemg has quit IRC (Ping timeout: 615 seconds)
03:41 πŸ”— Fusl chfoo: pushed a commit to the PR and commented
03:43 πŸ”— odemg has joined #urlteam
03:44 πŸ”— Fusl yeet
03:45 πŸ”— Fusl Flashfire: the party can begin i think
03:45 πŸ”— Flashfire Excellent
03:45 πŸ”— chfoo i'll have to update the warrior script and tracker
03:45 πŸ”— Fusl huh
03:45 πŸ”— Flashfire Alright then that shouldnt take too long though right?
03:45 πŸ”— Fusl chfoo: can you explain what needs to be updated?
03:45 πŸ”— kiska1 has joined #urlteam
03:46 πŸ”— chfoo update the submodule for warrior, then update the tracker so it shows the custom code warning
03:46 πŸ”— chfoo should take only a few minutes
03:46 πŸ”— Fusl oh are those things that cant be done with normal web ui admin access?
03:46 πŸ”— Fusl i mean he submodule update yeah
03:49 πŸ”— Fusl i'm trying to figure out how to streamline and document this process enough so that we can cut down on time from shortener discovery to full rollout of the custom code with all configurations set
03:50 πŸ”— TigerbotH has joined #urlteam
03:51 πŸ”— chfoo the tracker part is optional i think, but updating the submodule can be done by someone with access though
03:52 πŸ”— chfoo to the github repo, i.e. pull request
03:53 πŸ”— chfoo ok, done
03:53 πŸ”— chfoo oh, update the min version in the tracker so it doesn't try to use it without the custom code
03:54 πŸ”— Flashfire What is the mind version chfoo?
03:55 πŸ”— chfoo 52
03:59 πŸ”— Fusl Flashfire: oh and before you enable it
03:59 πŸ”— Fusl "get", not "head"
04:00 πŸ”— Fusl "head" will return you 405 Method Not Allowed
04:02 πŸ”— Flashfire I am currently on an unstable internet and while I would love to get the party started my siblings are pestering me to go over to the pool and the internet keeps dropping out. So I will keep that in mind and start it soon. Unless you want to laugh like a maniac and flip the switch
04:03 πŸ”— Fusl laughing like a maniac and flipping the switch is my daily driver
04:04 πŸ”— Fusl chfoo: everything looks goood i think?
04:05 πŸ”— chfoo i think so
04:05 πŸ”— Fusl set min version to 52, http method to get and redir status codes to 302 because thats what google uses
04:07 πŸ”— Fusl omgididit
04:12 πŸ”— Fusl hmm i think something is b0rk
04:13 πŸ”— Fusl terroroftinytown.client.errors.UnhandledStatusCode: Unknown status code 301 for 'http://goo.gl/k'
04:13 πŸ”— Fusl looks like they are /actually/ using 301s
04:13 πŸ”— Fusl huh
04:14 πŸ”— Flashfire Set up both of them as redirects?
04:14 πŸ”— Fusl yeah
04:14 πŸ”— Fusl lets see, this should work now
04:14 πŸ”— Flashfire I got on a bit more stable connection only problem I’m on mobile now lol
04:14 πŸ”— Fusl and indeed it does
04:15 πŸ”— Flashfire 170/170 found and scanned
04:15 πŸ”— Flashfire Nice lol
04:17 πŸ”— Fusl oh god no
04:17 πŸ”— Flashfire Fusl if you want to look at fascinating you can look at puri.na vs bit.ly both bitly URLs but over 40 short codes don’t match up so far
04:17 πŸ”— Fusl > http://goo.gl/{shortcode}
04:17 πŸ”— Fusl this shouldve been https
04:17 πŸ”— Flashfire Oh fuck
04:17 πŸ”— Flashfire hold on
04:18 πŸ”— Fusl http redirects to https
04:18 πŸ”— Fusl thats why we were seeing 100% found
04:18 πŸ”— Flashfire Oh shit
04:18 πŸ”— Flashfire I can’t find a box to turn it offπŸ€”
04:18 πŸ”— Fusl i did
04:19 πŸ”— Fusl thats also the reason we were seeing http 301
04:19 πŸ”— Fusl because i knew 301s arent used
04:19 πŸ”— Fusl but i never tested the http->https redirect
04:20 πŸ”— Fusl there, all fixed now
04:30 πŸ”— Flashfire Fusl I thought we agreed on a reduced queue to avoid a ban
04:32 πŸ”— Fusl Flashfire: you mean getting banned as a whole or per scraper?
04:32 πŸ”— Flashfire both
04:33 πŸ”— Fusl i dont think they give many fugs about us as a whole, same as they didnt for G+
04:33 πŸ”— Fusl and per scraper, 50 is so much below the threshold that triggers their rate limit
04:35 πŸ”— Fusl but tbh idc in the end, we can limit it back down to 10/5
04:41 πŸ”— Fusl Flashfire: lmk if you want me to set it down to 10/5
04:41 πŸ”— Flashfire Nah it should be fine if you are confident
04:46 πŸ”— Fusl 04:17 <@Flashfire> Fusl if you want to look at fascinating you can look at puri.na vs bit.ly both bitly URLs but over 40 short codes don’t match up so far
04:46 πŸ”— Fusl can you elaborate on that?
04:47 πŸ”— Flashfire bit.ly/1 vs puri.na/1 is the first example
04:47 πŸ”— Fusl yeah? they give different results
04:47 πŸ”— Flashfire Exactly most low level bitly urls dont do that
04:47 πŸ”— Flashfire they are both bitly urls
04:48 πŸ”— Fusl oh, as far as i know you can configure that
04:48 πŸ”— Fusl like, you can have a custom bitly domain and set it to resolve standard bitly stuff or dont
04:48 πŸ”— Fusl s/dont/not/
04:49 πŸ”— Fusl or at least you used to be able to do that
05:00 πŸ”— Fusl soon arriving at 4 char urls so we'll see more than 0-1 found per batch
05:02 πŸ”— Fusl actually nevermind, i think were doung uppercase chars still
05:02 πŸ”— Fusl yeah
05:11 πŸ”— Flashfire Fusl I am still keeping it at that low rate to avoid a ban
05:26 πŸ”— Flashfire rtsoftware.systems added to tracker however I am not sure I trust this connection I am on enough to make a big edit to the wiki
05:32 πŸ”— Fusl finally arrived at the end of 3 char urls, for real this time
05:33 πŸ”— Flashfire Hey Fusl if you want to add rtsoftware.systems to the URLteam warrior section that would be brilliant
05:34 πŸ”— Fusl "rtsoftware.systems". at this point url shorteners dont really make sense anymore LOL
05:35 πŸ”— Flashfire I mean I agree but someone made it
05:48 πŸ”— Fusl added, hopefully correct syntax but seems ok
05:48 πŸ”— Fusl (i really hate mediawiki lol)
05:51 πŸ”— Flashfire Yeah thats about right missing the pink bit but thats no huge biggie
05:53 πŸ”— Flashfire FUSL
05:53 πŸ”— Flashfire We getting rate limited
05:53 πŸ”— Flashfire 500
05:53 πŸ”— Flashfire Or maybe not
05:53 πŸ”— Flashfire a server error though
05:53 πŸ”— Fusl 500 are just server errors
05:53 πŸ”— Fusl everythings fine
05:53 πŸ”— Flashfire Ok
05:54 πŸ”— Flashfire Ill just add those back to the queue then
05:54 πŸ”— Fusl everything. is fine. http://xor.meo.ws/d5181c52/c797/4038/be34/3ced62acc345.png
05:54 πŸ”— Flashfire I am returning the claims to the queue that encountered server errors
05:54 πŸ”— Flashfire unless you want to edit the response codes for the 500s
05:55 πŸ”— Fusl nah, we shouldnt ignore 500s
05:55 πŸ”— Fusl and those return automatically in 30 minutes anyways i think
05:56 πŸ”— Flashfire Yeah they do but to speed it up sometimes I just do the releasing claims earlier if they error
05:56 πŸ”— Fusl well technically we can set the auto release to 10 minutes
05:57 πŸ”— Fusl right now items are flying in after around 45 seconds
05:57 πŸ”— Flashfire Fusl are you using IPV6?
05:57 πŸ”— Fusl nope
05:57 πŸ”— Flashfire Check the error reports for bitly_6
05:58 πŸ”— Fusl oof what the hell is that
05:58 πŸ”— Flashfire I dont know
06:16 πŸ”— Flashfire Fusl it doesnt look like the google thing is sorting itself out
06:18 πŸ”— Fusl Flashfire: it does do a great rate though
06:19 πŸ”— Fusl actually, it does not
06:19 πŸ”— Fusl huh
06:19 πŸ”— Flashfire Did it all get blocked up by the 500s?
06:19 πŸ”— Fusl seems so
06:20 πŸ”— Fusl i think including 500 in the banned status field will cause the client to retry 10 times
06:20 πŸ”— Fusl so we might be able to use that
07:08 πŸ”— Fusl Flashfire: looks like 500 errors are more than just temporary errors
07:09 πŸ”— Fusl https://goo.gl/1jJ3
07:09 πŸ”— Fusl this one for example throws a constant 500 error
07:10 πŸ”— Fusl so i'm thinking about adding 500 to the list of no redirect or unavailable status codes
07:10 πŸ”— Fusl chfoo?
07:19 πŸ”— Flashfire Sorry I was collecting fire wood Fusl I am not sure what difference it would make which one you added it to as long as it was no redirect or unavailable
07:19 πŸ”— Flashfire what were you doing to get rid of the errors before?
07:20 πŸ”— Flashfire please dont tell me it was delete job
07:20 πŸ”— Fusl delete job
07:20 πŸ”— Flashfire Fuck
07:20 πŸ”— Fusl nah jk, i clicked on delete error one each of em
07:21 πŸ”— Flashfire that means anything in that bracket of 50 wasnt grabbed
07:21 πŸ”— Fusl well at least before that
07:21 πŸ”— Fusl i havent deleted any errors since we are in the 4 char group
07:21 πŸ”— Fusl just lowered the auto requeue from 30m to 10m
07:21 πŸ”— Flashfire ok
07:22 πŸ”— Flashfire alright
07:22 πŸ”— Fusl imma add 500 to unavailable
07:22 πŸ”— Flashfire Ok
07:22 πŸ”— Flashfire wait so only some of the 500s permanent?
07:22 πŸ”— Flashfire Some were temporary?
07:23 πŸ”— Fusl all of them seem to be permanent
07:23 πŸ”— Fusl well
07:23 πŸ”— Fusl a very very very small percentage of them are temporary
07:23 πŸ”— Flashfire whats the numbers?
07:23 πŸ”— Fusl ec
07:23 πŸ”— Fusl sec
07:23 πŸ”— Fusl 2 in 164 500s were temporary
07:24 πŸ”— Flashfire hmm I think we can deal with those losses
07:25 πŸ”— Fusl chfoo: what do you think? i added 500 to unavailable status codes now so they are ignored. should we keep it like that or should we rather fail the items?
07:25 πŸ”— Fusl im gonna keep an eye on what the current 500s will do in a few hours, maybe they are temporary but just take longer to recover
07:32 πŸ”— Fusl so this, if im reading it correctly, should delete old errors every 5 mins https://github.com/ArchiveTeam/terroroftinytown/blob/1f5e121f85e734b71b06ebd69cf5131b8b8f51d3/terroroftinytown/tracker/app.py#L95-L111
07:32 πŸ”— Fusl by calling this function https://github.com/ArchiveTeam/terroroftinytown/blob/1f5e121f85e734b71b06ebd69cf5131b8b8f51d3/terroroftinytown/tracker/model.py#L521
07:32 πŸ”— Fusl which limits the delete to 1 error every 5 minutes
07:33 πŸ”— Fusl which seems to be correct since one error just disappeared from the dashboard
07:33 πŸ”— Fusl yay i can read code
07:42 πŸ”— Fusl i'm sorry i had to https://atdash.meo.ws/d/urlteam/urlteam-stats?from=now-30m
07:48 πŸ”— Zerote has joined #urlteam
08:18 πŸ”— Flashfire terroroftinytown.client.errors.UnhandledStatusCode: Unknown status code 530 for 'https://is.gd/1CazbD'
08:18 πŸ”— Flashfire what the fuck is a 530
08:20 πŸ”— Flashfire Oh fuck
08:20 πŸ”— Flashfire Is.gd may have enabled cloudflare
08:20 πŸ”— Flashfire FUSL CHFOO SOMEBODY2 JAA KAZ HOOK54321
08:20 πŸ”— Flashfire can any of you confirm this?
08:20 πŸ”— Fusl its behind CF but that url is giving a 404 here
08:21 πŸ”— hook54321 no need to ping everyone though :)
08:21 πŸ”— Fusl > Error 1016 / Error 530 indicates Cloudflare is unable to send requests to your server because its origin IP cannot resolve the A or CNAME DNS record requested.
08:23 πŸ”— Flashfire I am turning off autoqueue for the moment
08:23 πŸ”— Flashfire I dont think it was behind cloud flare before
08:24 πŸ”— Fusl its been behind CF since 2017-05-26
08:25 πŸ”— Flashfire ok then
08:25 πŸ”— Flashfire I am bad at research thats a given
08:25 πŸ”— Flashfire but the error of 530 I have no idea what threw it
08:25 πŸ”— Flashfire Or these IPV6 errors on bitly_6
08:27 πŸ”— Fusl been trying to figure that out since you mentioned that earlier
08:27 πŸ”— Fusl but cant seem to reproduce this
08:29 πŸ”— Flashfire No idea either cause I cant read code
08:46 πŸ”— Fusl yeah absolutely no idea whats happening there
09:24 πŸ”— seatsea has quit IRC (Read error: Connection reset by peer)
09:39 πŸ”— Fusl still no luck in reproducing this
09:40 πŸ”— Fusl im just giving up at this point
11:10 πŸ”— seatsea has joined #urlteam
11:17 πŸ”— seatsea has quit IRC (Quit: The Lounge - https://thelounge.github.io)
11:18 πŸ”— JAA Re IPv6: I haven't looked at it, but could it be misconfigured machines which have IPv6 enabled and resolve AAAA records but can't route IPv6 traffic?
11:19 πŸ”— seatsea has joined #urlteam
11:26 πŸ”— Fusl JAA: bit.ly doesnt have ipv6 records
11:27 πŸ”— JAA Right
11:28 πŸ”— JAA What's the error? Can't see any on the tracker right now.
11:30 πŸ”— Fusl "ValueError: Invalid IPv6 URL"
11:32 πŸ”— Fusl in main.py fetch_url line 86, the call to `requests.head`
12:22 πŸ”— seatsea has quit IRC (Quit: The Lounge - https://thelounge.github.io)
14:13 πŸ”— seatsea has joined #urlteam
14:13 πŸ”— seatsea has quit IRC (Client Quit)
15:08 πŸ”— seatsea has joined #urlteam
15:14 πŸ”— Fusl < location: http://www.google.com/sorry/media/players/not/allowed/
15:14 πŸ”— Fusl thats a new one lol
18:25 πŸ”— ivan has quit IRC (Read error: Operation timed out)
18:25 πŸ”— JAA has quit IRC (Read error: Operation timed out)
18:25 πŸ”— Mayonaise has quit IRC (Read error: Operation timed out)
18:25 πŸ”— ivan has joined #urlteam
18:25 πŸ”— Mayonaise has joined #urlteam
18:26 πŸ”— lunik1 has quit IRC (Read error: Operation timed out)
19:26 πŸ”— lunik1 has joined #urlteam
19:29 πŸ”— JAA has joined #urlteam
19:29 πŸ”— bakJAA sets mode: +o JAA
19:29 πŸ”— Fusl sets mode: +o JAA
20:25 πŸ”— chfoo is everything ok?
20:45 πŸ”— Zerote has quit IRC (Ping timeout: 260 seconds)
21:07 πŸ”— Zerote has joined #urlteam
22:19 πŸ”— Jens has quit IRC (Remote host closed the connection)
22:20 πŸ”— Jens has joined #urlteam

irclogger-viewer