[18:00] chfoo: Did you have any troubles with bans from visibli? When I tried it on a server, it got banned and the ban never expires. I didn't add any more tasks to the server for visibli since I don't want to get users banned [18:47] soultcer: i havn't gotten any bans since my last commit to fix up the script. right now i'm grabbing the hex codes at a very slow rate through tor and using a list common user agents. [18:50] the delay between requests is based on the sine function and it averages a little above 1 url per second [18:53] Nice [18:58] i have no idea how to grab the regular shortcodes at a practical rate without causing noticeable traffic though [22:22] larger botnet [22:46] the number of machines doesn't really matter. it's more like salami slicing. [22:47] the goal is to get all the urls using the least amount of traffic for both parties [22:54] an idea i have so far is to search through google, twitter etc for valid shortcodes since they have an automated banning system [22:56] it might be possible that the shortcodes arn't random too [22:57] chfoo, some work has been done in this. Some of the url shorteners we just increment and test [22:57] I thought we were scraping twitter for urls but I was told that isn't happening. I know we have the twitter data to do it [23:02] i havn't experimented much with their modern shortcodes system yet so it's just a black box to me right now. [23:03] the main page and wiki have a lot info we put [23:04] Also some shorteners are just aliases to other services which means the hashes are the same [23:04] which is nice [23:22] oh cool, i didn't know about the twitter data archives before. [23:24] for now i'll keep grabbing visibli hex. then when i have more time, i'll investigate a bit more with normal visibli.