[00:31] *** bwn has quit IRC (Read error: Operation timed out) [00:54] *** bwn has joined #urlteam [02:01] *** aaaaaaaaa has quit IRC (Leaving) [02:12] *** JesseW has joined #urlteam [02:17] *** aaaaaaaaa has joined #urlteam [02:17] *** swebb sets mode: +o aaaaaaaaa [03:35] *** bwn has quit IRC (Read error: Operation timed out) [03:46] *** JW_work has joined #urlteam [03:46] *** JW_work1 has quit IRC (Read error: Connection reset by peer) [03:51] *** svchfoo1 has quit IRC (Ping timeout: 369 seconds) [04:14] *** svchfoo1 has joined #urlteam [04:15] *** svchfoo3 sets mode: +o svchfoo1 [04:20] *** atlogbot has quit IRC (Ping timeout: 369 seconds) [04:21] *** svchfoo1 has quit IRC (Ping timeout: 369 seconds) [04:23] *** Domin- has joined #urlteam [04:27] *** Domin_ has quit IRC (Read error: Operation timed out) [04:29] *** aaaaaaaaa has quit IRC (Leaving) [04:43] *** chazchaz has quit IRC (Ping timeout: 369 seconds) [04:46] *** atlogbot has joined #urlteam [04:46] *** svchfoo1 has joined #urlteam [04:47] *** svchfoo3 sets mode: +o svchfoo1 [04:49] *** chazchaz has joined #urlteam [05:06] *** swebb has quit IRC (hub.efnet.us irc.colosolutions.net) [05:19] *** swebb_ has joined #urlteam [05:20] How can I figure out what HTTP status codes a shortner uses for "unavailable" and "banned"? Specifically, the http://www.migre.me shortner... [05:22] *** swebb_ is now known as swebb [05:23] *** svchfoo3 sets mode: +o swebb [05:33] *** Deewiant has quit IRC (hub.se irc.du.se) [08:00] *** bwn has joined #urlteam [08:52] *** JesseW has quit IRC (Leaving.) [09:48] *** bwn has quit IRC (Ping timeout: 483 seconds) [11:03] *** bwn has joined #urlteam [12:47] *** WinterFox has joined #urlteam [12:53] *** VADemon has joined #urlteam [13:47] *** Deewiant has joined #urlteam [14:06] *** WinterFox has quit IRC (Remote host closed the connection) [15:16] *** Start has quit IRC (Quit: Disconnected.) [16:07] *** Start has joined #urlteam [16:37] *** Start has quit IRC (Quit: Disconnected.) [16:42] *** Start has joined #urlteam [17:02] *** Start has quit IRC (Quit: Disconnected.) [17:04] *** JesseW has joined #urlteam [17:16] There is a code for limit exceeded, I believe [17:18] 429 [17:29] swebb: was that intended as an answer to my question? [17:29] Yea. [17:30] then I don't understand it. :-) [17:30] Each shortener can have their own behaviour and code though... [17:30] The urlteam tracker however, responds with HTTP 429 when it thinks you're annoying IIRC [17:31] ah, yes. I'm not asking about the urltream tracker, but about a specific shortner that I'm hoping to add [17:31] http://httpstatusdogs.com/429-too-many-requests [17:31] I suppose I can just start without anything in those fields (or the default in banned) and see how it goes [17:31] Many use 429, but you can't assume that every one does. [17:32] Just time out or handle non-redirects all the same. [17:32] so you only have to worry about a few different cases and throw the rest away or leave as unused. [17:33] nods. well, then I have a project ready to go -- but I don't want to start it off without having chfoo (or arkiver) look it over and make sure I'm not going to break things [17:33] * JesseW needs to go AFK for a while now anyway [17:34] *** JesseW has quit IRC (Leaving.) [17:34] and might be worth doing parallel runs and compare the results for some known keys etc [17:34] sometimes you can get different results back, depending on if they currently hate you or not [17:34] *** bwn has quit IRC (Read error: Operation timed out) [18:05] *** aaaaaaaaa has joined #urlteam [18:05] *** swebb sets mode: +o aaaaaaaaa [18:05] *** Start has joined #urlteam [18:20] *** Start has quit IRC (Quit: Disconnected.) [18:26] *** Start has joined #urlteam [18:39] *** Start has quit IRC (Quit: Disconnected.) [18:45] *** Start has joined #urlteam [19:06] *** Start has quit IRC (Quit: Disconnected.) [20:07] *** dashcloud has quit IRC (Read error: Operation timed out) [20:11] *** dashcloud has joined #urlteam [20:11] *** svchfoo1 sets mode: +o dashcloud [20:48] *** bwn has joined #urlteam [20:49] *** Start has joined #urlteam [20:53] *** aaaaaaaa_ has joined #urlteam [20:53] *** aaaaaaaaa has quit IRC (Read error: Connection reset by peer) [20:53] *** swebb sets mode: +o aaaaaaaa_ [20:53] *** aaaaaaaa_ is now known as aaaaaaaaa [21:26] *** slang has joined #urlteam [21:30] *** JW_work1 has joined #urlteam [21:31] *** Start has quit IRC (Quit: Disconnected.) [21:31] *** JW_work has quit IRC (Read error: Operation timed out) [21:55] *** WinterFox has joined #urlteam [21:55] Why are there no updates to the torrent file? [22:17] WinterFox: the new scraper (run through the Warrior) outputs in separate, daily items uploaded to the Internet Archive [22:18] e.g. https://archive.org/details/urlteam_2015-05-22-00-26-03 [22:19] you can download them via bittorrent, but there's not just one, changing torrent [22:19] Is there any way to mass download it all from the internet archive? [22:20] automatically, no — I hacked together various scripts to do so. The names are consistent, so it's not that difficult... [22:20] it would probably be good to make big, yearly combined items [22:21] I'll try to post my hacked up scripts soon [23:07] *** JW_work1 has quit IRC (Quit: Leaving.) [23:10] *** JW_work has joined #urlteam [23:30] *** JW_work has quit IRC (Leaving.) [23:31] *** JW_work has joined #urlteam [23:31] *** aaaaaaaaa has quit IRC (Ping timeout: 615 seconds) [23:32] *** JW_work has quit IRC (Client Quit) [23:41] *** JW_work has joined #urlteam [23:42] *** JW_work has quit IRC (Client Quit) [23:43] *** JW_work has joined #urlteam [23:43] *** aaaaaaaaa has joined #urlteam [23:43] *** swebb sets mode: +o aaaaaaaaa [23:47] *** JW_work has quit IRC (Client Quit) [23:47] *** JW_work has joined #urlteam [23:50] *** JW_work has quit IRC (Client Quit) [23:58] *** JW_work has joined #urlteam [23:58] *** JW_work has quit IRC (Client Quit)