[00:00] everyone: see question in #archiveteam [00:19] *** xmc sets mode: +oooo chfoo Sanqui SketchCow Frogging [00:19] *** xmc sets mode: +oo swebb godane [00:20] *** xmc sets mode: +ooo DFJustin Asparagir closure [00:21] *** xmc sets mode: +o yipdw [00:22] wonder what happened [00:22] netsplits be crazy as of late [00:25] efnet is rotting, slowly [01:07] i336_: please ask that in #archiveteam-bs next time [01:15] arkiver: sorry. sure thing [01:24] i336_: why is 16 minutes so bad [01:24] yipdw: let's move to #archiveteam-bs [01:24] ....we're already there. I didn't see. [01:24] we are alread- [01:24] where do you think this is [01:24] sorry [01:24] anyway, it's 16 minutes or you spend 19 days wondering how you could be faster and end up with nothing [01:24] you're spending far more than 16 minutes overthinking how to do it faster [01:25] this is 16 minutes per search result, and if we do more than one search at a time that's 16*(number of searches in progress) for your results to come back [01:25] you should plan speedups while you already have the slow script working in the background [01:25] this is for finding content to save manually [01:25] I was hoping for something fast [01:26] research what the exact ratelimit is and aim for ~80-95% of it [01:26] if they won't tell you, go with a half-second and watch the error rate [01:26] [Project log] "Well, I found the ratelimit, but now I need a new IP address." [01:26] a lot of APIs will tell you what your ratelimit is per unit time [01:27] do you need a new IP address, or do you just need to back off for some amount of time? [01:27] if you go *too* fast it wouldn't surprise me if you get blocked for a longer period [01:27] yipdw: this isn't like an oauth type thing. it just returns results. there's no measurement. this is a forgotten API they forgot to turn off... so it's a fine line between "nobody will realize" and "OOPS WE FORGOT TO--" *pulls the plug* [01:28] which is Bad(TM) because ex.ua go baibai on the 31st [01:28] what does OAuth have to do with this? [01:28] OAuth and ratelimiting are independent [01:28] i336_: do you have the crap-that-takes-16-minutes already running right now? [01:29] nicolas17: arkiver is currently working on crawling the site, once that comes back, we can just search the local mirror [01:29] let's keep everything about this project in #exexbaby [01:29] okay. [01:36] *** BartoCH has quit IRC (Quit: WeeChat 1.6) [01:39] *** BartoCH has joined #archiveteam-bs [01:48] *** ZizzyDizz has joined #archiveteam-bs [01:49] *** ZizzyDizz has quit IRC (Client Quit) [02:05] *** VADemon has quit IRC (Quit: left4dead) [02:35] *** Asparagir has quit IRC (Asparagir) [02:37] *** kristian_ has quit IRC (Quit: Leaving) [03:21] *** Asparagir has joined #archiveteam-bs [04:11] *** dashcloud has quit IRC (Read error: Operation timed out) [04:15] *** dashcloud has joined #archiveteam-bs [05:14] *** vitzli has joined #archiveteam-bs [05:21] *** Stiletto has joined #archiveteam-bs [05:36] *** Sk1d has quit IRC (Ping timeout: 194 seconds) [05:42] *** Sk1d has joined #archiveteam-bs [06:37] *** Start_ has quit IRC (Quit: Disconnected.) [06:37] *** Start has joined #archiveteam-bs [06:43] *** nicolas17 has quit IRC (Quit: nuff 4 2day) [07:07] *** jspiros has quit IRC (Read error: Operation timed out) [07:08] *** jspiros has joined #archiveteam-bs [07:21] *** vitzli has quit IRC (Quit: Leaving) [07:25] *** vitzli has joined #archiveteam-bs [07:42] *** ravetcofx has quit IRC (Read error: Operation timed out) [08:11] *** krazedkat has quit IRC (Ping timeout: 244 seconds) [08:17] *** SadDM has quit IRC (Read error: Operation timed out) [08:17] *** SadDM has joined #archiveteam-bs [08:17] *** swebb sets mode: +o SadDM [08:32] *** SadDM has quit IRC (Read error: Operation timed out) [08:40] *** SadDM has joined #archiveteam-bs [08:40] *** swebb sets mode: +o SadDM [08:52] *** GE has joined #archiveteam-bs [09:02] *** SadDM has quit IRC (Read error: Operation timed out) [09:05] *** SadDM has joined #archiveteam-bs [09:05] *** swebb sets mode: +o SadDM [09:32] *** SadDM has quit IRC (Read error: Operation timed out) [09:35] *** SadDM has joined #archiveteam-bs [09:35] *** swebb sets mode: +o SadDM [09:41] *** SadDM has quit IRC (Read error: Operation timed out) [09:47] *** SadDM has joined #archiveteam-bs [09:47] *** swebb sets mode: +o SadDM [09:49] *** dashcloud has quit IRC (Read error: Operation timed out) [09:53] *** dashcloud has joined #archiveteam-bs [10:00] *** SadDM has quit IRC (Read error: Operation timed out) [10:02] *** SadDM has joined #archiveteam-bs [10:02] *** swebb sets mode: +o SadDM [10:07] *** SadDM has quit IRC (Read error: Operation timed out) [10:08] *** SadDM has joined #archiveteam-bs [10:08] *** swebb sets mode: +o SadDM [10:11] *** BlueMaxim has quit IRC (Quit: Leaving) [11:06] *** krazedkat has joined #archiveteam-bs [11:47] *** BartoCH has quit IRC (Ping timeout: 260 seconds) [11:50] *** BartoCH has joined #archiveteam-bs [12:04] *** GE has quit IRC (Remote host closed the connection) [12:45] *** Asparagir has quit IRC (Read error: Operation timed out) [12:56] *** Asparagir has joined #archiveteam-bs [13:31] *** GE has joined #archiveteam-bs [13:51] *** fie has quit IRC (Ping timeout: 506 seconds) [14:15] *** i336_ has quit IRC (Ping timeout: 260 seconds) [14:53] *** vitzli has quit IRC (Quit: Leaving) [14:59] *** dashcloud has quit IRC (Read error: Operation timed out) [15:04] *** dashcloud has joined #archiveteam-bs [15:42] *** fie has joined #archiveteam-bs [15:57] *** fie has quit IRC (Read error: Operation timed out) [16:51] i'm grabbing descriptions of the Rush Limbaugh show going back 6 years [16:55] *** dashcloud has quit IRC (Read error: Operation timed out) [16:59] *** dashcloud has joined #archiveteam-bs [17:07] *** HCross has quit IRC (Read error: Operation timed out) [17:13] *** HarryCros has joined #archiveteam-bs [17:21] *** ndiddy has joined #archiveteam-bs [17:43] *** fie has joined #archiveteam-bs [17:47] *** nicolas17 has joined #archiveteam-bs [18:31] SketchCow: I saw you moved my Yahoo Groups crawl into the archiveteam/web collection. Given the number of items created so far, would it make sense to create a separate collection just for this data? With proper permissions I could organize new uploads into that new collection myself. [18:52] *** Rye has quit IRC (Quit: ZNC - http://znc.in) [18:55] *** Rye has joined #archiveteam-bs [18:55] *** Rye has quit IRC (Remote host closed the connection) [18:57] *** Rye has joined #archiveteam-bs [19:04] *** Rye has quit IRC (Quit: ZNC - http://znc.in) [19:08] *** Rye has joined #archiveteam-bs [19:26] *** brayden has quit IRC (Ping timeout: 633 seconds) [20:47] *** whopper has quit IRC (hub.se irc.efnet.nl) [20:47] *** zerkalo has quit IRC (hub.se irc.efnet.nl) [20:47] *** wacky has quit IRC (hub.se irc.efnet.nl) [20:47] *** luckcolor has quit IRC (hub.se irc.efnet.nl) [20:47] *** w0pr has joined #archiveteam-bs [20:47] *** zerkalo_ has joined #archiveteam-bs [20:52] *** wacky_ has joined #archiveteam-bs [21:03] *** luckcolor has joined #archiveteam-bs [21:16] *** RichardG_ has joined #archiveteam-bs [21:19] *** RichardG has quit IRC (Read error: Operation timed out) [22:05] *** t2t2 has quit IRC (Ping timeout: 260 seconds) [22:32] *** BlueMaxim has joined #archiveteam-bs [22:43] *** yipdw has quit IRC (Quit: yipdw) [22:44] *** yipdw has joined #archiveteam-bs [22:44] *** Frogging sets mode: +o yipdw [22:50] *** GE has quit IRC (Quit: zzz) [22:58] *** t2t2 has joined #archiveteam-bs [23:51] *** i336_ has joined #archiveteam-bs