#urlteam 2012-12-08,Sat

↑back Search

Time Nickname Message
18:22 🔗 deathy so how do new available tasks actually get generated/found?
18:37 🔗 soultcer deathy: I generate them when there are only few left. For example on tinyurl, I am doing a complete sequential scan for all codes up to 6 letters
18:38 🔗 soultcer On bitly or isgd I just let it check random codes with 6 (bitly) or 5 (isgd) characters
18:39 🔗 soultcer owly and ur1ca are sequential as well. With owly we are still catching up, with ur1ca we are already caught up. For both of them we are fetching shorturls faster than people are creating them
18:42 🔗 deathy that's interesting. no kind of scraping other things (twitter stream, other archiving efforts) for shortener links? (though no idea if that might be useful)
18:43 🔗 soultcer swebb is fetching all shorturls from the twitter "spritzer" (a feed containing about 2% of all tweets)
18:49 🔗 deathy cool
18:49 🔗 deathy obviously this is a never-ending project :)
18:52 🔗 soultcer Yes, unless all shorteners suddenly decide to shut down :D
19:09 🔗 GitHub29 [tinyback] soult pushed 3 new commits to master: https://github.com/soult/tinyback/compare/d49d8ef80688...588c45c9fe83
19:09 🔗 GitHub29 tinyback/master 588c45c David Triendl: Bump version to 2.3
19:09 🔗 GitHub29 tinyback/master 5f1da45 David Triendl: services.Tinyurl: Try to get original URL from affiliate URL
19:09 🔗 GitHub29 tinyback/master 7a6a820 David Triendl: Merge branch 'add-snipurl-service'
19:10 🔗 soultcer ersi: I have set the rate limit for snipurl to 2 requests / second for now. I rather stay on the safe side than getting all scrapers banned
19:17 🔗 deathy forgot to ask... any rules on multiple threads for tinyback? since requests/second limit and things...
19:21 🔗 soultcer deathy: Nope, the tracker will take care of it. It automatically limits you to one thread per url shortener for each IP address
19:24 🔗 deathy k. good to know
23:31 🔗 ersi soultcer: Yeah, sounds good

irclogger-viewer