#archiveteam-bs 2016-12-11,Sun

↑back Search

Time Nickname Message
00:00 🔗 i336_ everyone: see question in #archiveteam
00:19 🔗 xmc sets mode: +oooo chfoo Sanqui SketchCow Frogging
00:19 🔗 xmc sets mode: +oo swebb godane
00:20 🔗 xmc sets mode: +ooo DFJustin Asparagir closure
00:21 🔗 xmc sets mode: +o yipdw
00:22 🔗 xmc wonder what happened
00:22 🔗 vantec netsplits be crazy as of late
00:25 🔗 xmc efnet is rotting, slowly
01:07 🔗 arkiver i336_: please ask that in #archiveteam-bs next time
01:15 🔗 i336_ arkiver: sorry. sure thing
01:24 🔗 yipdw i336_: why is 16 minutes so bad
01:24 🔗 i336_ yipdw: let's move to #archiveteam-bs
01:24 🔗 i336_ ....we're already there. I didn't see.
01:24 🔗 Frogging we are alread-
01:24 🔗 yipdw where do you think this is
01:24 🔗 i336_ sorry
01:24 🔗 yipdw anyway, it's 16 minutes or you spend 19 days wondering how you could be faster and end up with nothing
01:24 🔗 nicolas17 you're spending far more than 16 minutes overthinking how to do it faster
01:25 🔗 i336_ this is 16 minutes per search result, and if we do more than one search at a time that's 16*(number of searches in progress) for your results to come back
01:25 🔗 nicolas17 you should plan speedups while you already have the slow script working in the background
01:25 🔗 i336_ this is for finding content to save manually
01:25 🔗 i336_ I was hoping for something fast
01:26 🔗 yipdw research what the exact ratelimit is and aim for ~80-95% of it
01:26 🔗 yipdw if they won't tell you, go with a half-second and watch the error rate
01:26 🔗 i336_ [Project log] "Well, I found the ratelimit, but now I need a new IP address."
01:26 🔗 yipdw a lot of APIs will tell you what your ratelimit is per unit time
01:27 🔗 yipdw do you need a new IP address, or do you just need to back off for some amount of time?
01:27 🔗 nicolas17 if you go *too* fast it wouldn't surprise me if you get blocked for a longer period
01:27 🔗 i336_ yipdw: this isn't like an oauth type thing. it just returns results. there's no measurement. this is a forgotten API they forgot to turn off... so it's a fine line between "nobody will realize" and "OOPS WE FORGOT TO--" *pulls the plug*
01:28 🔗 i336_ which is Bad(TM) because ex.ua go baibai on the 31st
01:28 🔗 yipdw what does OAuth have to do with this?
01:28 🔗 yipdw OAuth and ratelimiting are independent
01:28 🔗 nicolas17 i336_: do you have the crap-that-takes-16-minutes already running right now?
01:29 🔗 i336_ nicolas17: arkiver is currently working on crawling the site, once that comes back, we can just search the local mirror
01:29 🔗 arkiver let's keep everything about this project in #exexbaby
01:29 🔗 i336_ okay.
01:36 🔗 BartoCH has quit IRC (Quit: WeeChat 1.6)
01:39 🔗 BartoCH has joined #archiveteam-bs
01:48 🔗 ZizzyDizz has joined #archiveteam-bs
01:49 🔗 ZizzyDizz has quit IRC (Client Quit)
02:05 🔗 VADemon has quit IRC (Quit: left4dead)
02:35 🔗 Asparagir has quit IRC (Asparagir)
02:37 🔗 kristian_ has quit IRC (Quit: Leaving)
03:21 🔗 Asparagir has joined #archiveteam-bs
04:11 🔗 dashcloud has quit IRC (Read error: Operation timed out)
04:15 🔗 dashcloud has joined #archiveteam-bs
05:14 🔗 vitzli has joined #archiveteam-bs
05:21 🔗 Stiletto has joined #archiveteam-bs
05:36 🔗 Sk1d has quit IRC (Ping timeout: 194 seconds)
05:42 🔗 Sk1d has joined #archiveteam-bs
06:37 🔗 Start_ has quit IRC (Quit: Disconnected.)
06:37 🔗 Start has joined #archiveteam-bs
06:43 🔗 nicolas17 has quit IRC (Quit: nuff 4 2day)
07:07 🔗 jspiros has quit IRC (Read error: Operation timed out)
07:08 🔗 jspiros has joined #archiveteam-bs
07:21 🔗 vitzli has quit IRC (Quit: Leaving)
07:25 🔗 vitzli has joined #archiveteam-bs
07:42 🔗 ravetcofx has quit IRC (Read error: Operation timed out)
08:11 🔗 krazedkat has quit IRC (Ping timeout: 244 seconds)
08:17 🔗 SadDM has quit IRC (Read error: Operation timed out)
08:17 🔗 SadDM has joined #archiveteam-bs
08:17 🔗 swebb sets mode: +o SadDM
08:32 🔗 SadDM has quit IRC (Read error: Operation timed out)
08:40 🔗 SadDM has joined #archiveteam-bs
08:40 🔗 swebb sets mode: +o SadDM
08:52 🔗 GE has joined #archiveteam-bs
09:02 🔗 SadDM has quit IRC (Read error: Operation timed out)
09:05 🔗 SadDM has joined #archiveteam-bs
09:05 🔗 swebb sets mode: +o SadDM
09:32 🔗 SadDM has quit IRC (Read error: Operation timed out)
09:35 🔗 SadDM has joined #archiveteam-bs
09:35 🔗 swebb sets mode: +o SadDM
09:41 🔗 SadDM has quit IRC (Read error: Operation timed out)
09:47 🔗 SadDM has joined #archiveteam-bs
09:47 🔗 swebb sets mode: +o SadDM
09:49 🔗 dashcloud has quit IRC (Read error: Operation timed out)
09:53 🔗 dashcloud has joined #archiveteam-bs
10:00 🔗 SadDM has quit IRC (Read error: Operation timed out)
10:02 🔗 SadDM has joined #archiveteam-bs
10:02 🔗 swebb sets mode: +o SadDM
10:07 🔗 SadDM has quit IRC (Read error: Operation timed out)
10:08 🔗 SadDM has joined #archiveteam-bs
10:08 🔗 swebb sets mode: +o SadDM
10:11 🔗 BlueMaxim has quit IRC (Quit: Leaving)
11:06 🔗 krazedkat has joined #archiveteam-bs
11:47 🔗 BartoCH has quit IRC (Ping timeout: 260 seconds)
11:50 🔗 BartoCH has joined #archiveteam-bs
12:04 🔗 GE has quit IRC (Remote host closed the connection)
12:45 🔗 Asparagir has quit IRC (Read error: Operation timed out)
12:56 🔗 Asparagir has joined #archiveteam-bs
13:31 🔗 GE has joined #archiveteam-bs
13:51 🔗 fie has quit IRC (Ping timeout: 506 seconds)
14:15 🔗 i336_ has quit IRC (Ping timeout: 260 seconds)
14:53 🔗 vitzli has quit IRC (Quit: Leaving)
14:59 🔗 dashcloud has quit IRC (Read error: Operation timed out)
15:04 🔗 dashcloud has joined #archiveteam-bs
15:42 🔗 fie has joined #archiveteam-bs
15:57 🔗 fie has quit IRC (Read error: Operation timed out)
16:51 🔗 godane i'm grabbing descriptions of the Rush Limbaugh show going back 6 years
16:55 🔗 dashcloud has quit IRC (Read error: Operation timed out)
16:59 🔗 dashcloud has joined #archiveteam-bs
17:07 🔗 HCross has quit IRC (Read error: Operation timed out)
17:13 🔗 HarryCros has joined #archiveteam-bs
17:21 🔗 ndiddy has joined #archiveteam-bs
17:43 🔗 fie has joined #archiveteam-bs
17:47 🔗 nicolas17 has joined #archiveteam-bs
18:31 🔗 PurpleSym SketchCow: I saw you moved my Yahoo Groups crawl into the archiveteam/web collection. Given the number of items created so far, would it make sense to create a separate collection just for this data? With proper permissions I could organize new uploads into that new collection myself.
18:52 🔗 Rye has quit IRC (Quit: ZNC - http://znc.in)
18:55 🔗 Rye has joined #archiveteam-bs
18:55 🔗 Rye has quit IRC (Remote host closed the connection)
18:57 🔗 Rye has joined #archiveteam-bs
19:04 🔗 Rye has quit IRC (Quit: ZNC - http://znc.in)
19:08 🔗 Rye has joined #archiveteam-bs
19:26 🔗 brayden has quit IRC (Ping timeout: 633 seconds)
20:47 🔗 whopper has quit IRC (hub.se irc.efnet.nl)
20:47 🔗 zerkalo has quit IRC (hub.se irc.efnet.nl)
20:47 🔗 wacky has quit IRC (hub.se irc.efnet.nl)
20:47 🔗 luckcolor has quit IRC (hub.se irc.efnet.nl)
20:47 🔗 w0pr has joined #archiveteam-bs
20:47 🔗 zerkalo_ has joined #archiveteam-bs
20:52 🔗 wacky_ has joined #archiveteam-bs
21:03 🔗 luckcolor has joined #archiveteam-bs
21:16 🔗 RichardG_ has joined #archiveteam-bs
21:19 🔗 RichardG has quit IRC (Read error: Operation timed out)
22:05 🔗 t2t2 has quit IRC (Ping timeout: 260 seconds)
22:32 🔗 BlueMaxim has joined #archiveteam-bs
22:43 🔗 yipdw has quit IRC (Quit: yipdw)
22:44 🔗 yipdw has joined #archiveteam-bs
22:44 🔗 Frogging sets mode: +o yipdw
22:50 🔗 GE has quit IRC (Quit: zzz)
22:58 🔗 t2t2 has joined #archiveteam-bs
23:51 🔗 i336_ has joined #archiveteam-bs

irclogger-viewer