| Time |
Nickname |
Message |
|
00:00
🔗
|
i336_ |
everyone: see question in #archiveteam |
|
00:19
🔗
|
|
xmc sets mode: +oooo chfoo Sanqui SketchCow Frogging |
|
00:19
🔗
|
|
xmc sets mode: +oo swebb godane |
|
00:20
🔗
|
|
xmc sets mode: +ooo DFJustin Asparagir closure |
|
00:21
🔗
|
|
xmc sets mode: +o yipdw |
|
00:22
🔗
|
xmc |
wonder what happened |
|
00:22
🔗
|
vantec |
netsplits be crazy as of late |
|
00:25
🔗
|
xmc |
efnet is rotting, slowly |
|
01:07
🔗
|
arkiver |
i336_: please ask that in #archiveteam-bs next time |
|
01:15
🔗
|
i336_ |
arkiver: sorry. sure thing |
|
01:24
🔗
|
yipdw |
i336_: why is 16 minutes so bad |
|
01:24
🔗
|
i336_ |
yipdw: let's move to #archiveteam-bs |
|
01:24
🔗
|
i336_ |
....we're already there. I didn't see. |
|
01:24
🔗
|
Frogging |
we are alread- |
|
01:24
🔗
|
yipdw |
where do you think this is |
|
01:24
🔗
|
i336_ |
sorry |
|
01:24
🔗
|
yipdw |
anyway, it's 16 minutes or you spend 19 days wondering how you could be faster and end up with nothing |
|
01:24
🔗
|
nicolas17 |
you're spending far more than 16 minutes overthinking how to do it faster |
|
01:25
🔗
|
i336_ |
this is 16 minutes per search result, and if we do more than one search at a time that's 16*(number of searches in progress) for your results to come back |
|
01:25
🔗
|
nicolas17 |
you should plan speedups while you already have the slow script working in the background |
|
01:25
🔗
|
i336_ |
this is for finding content to save manually |
|
01:25
🔗
|
i336_ |
I was hoping for something fast |
|
01:26
🔗
|
yipdw |
research what the exact ratelimit is and aim for ~80-95% of it |
|
01:26
🔗
|
yipdw |
if they won't tell you, go with a half-second and watch the error rate |
|
01:26
🔗
|
i336_ |
[Project log] "Well, I found the ratelimit, but now I need a new IP address." |
|
01:26
🔗
|
yipdw |
a lot of APIs will tell you what your ratelimit is per unit time |
|
01:27
🔗
|
yipdw |
do you need a new IP address, or do you just need to back off for some amount of time? |
|
01:27
🔗
|
nicolas17 |
if you go *too* fast it wouldn't surprise me if you get blocked for a longer period |
|
01:27
🔗
|
i336_ |
yipdw: this isn't like an oauth type thing. it just returns results. there's no measurement. this is a forgotten API they forgot to turn off... so it's a fine line between "nobody will realize" and "OOPS WE FORGOT TO--" *pulls the plug* |
|
01:28
🔗
|
i336_ |
which is Bad(TM) because ex.ua go baibai on the 31st |
|
01:28
🔗
|
yipdw |
what does OAuth have to do with this? |
|
01:28
🔗
|
yipdw |
OAuth and ratelimiting are independent |
|
01:28
🔗
|
nicolas17 |
i336_: do you have the crap-that-takes-16-minutes already running right now? |
|
01:29
🔗
|
i336_ |
nicolas17: arkiver is currently working on crawling the site, once that comes back, we can just search the local mirror |
|
01:29
🔗
|
arkiver |
let's keep everything about this project in #exexbaby |
|
01:29
🔗
|
i336_ |
okay. |
|
01:36
🔗
|
|
BartoCH has quit IRC (Quit: WeeChat 1.6) |
|
01:39
🔗
|
|
BartoCH has joined #archiveteam-bs |
|
01:48
🔗
|
|
ZizzyDizz has joined #archiveteam-bs |
|
01:49
🔗
|
|
ZizzyDizz has quit IRC (Client Quit) |
|
02:05
🔗
|
|
VADemon has quit IRC (Quit: left4dead) |
|
02:35
🔗
|
|
Asparagir has quit IRC (Asparagir) |
|
02:37
🔗
|
|
kristian_ has quit IRC (Quit: Leaving) |
|
03:21
🔗
|
|
Asparagir has joined #archiveteam-bs |
|
04:11
🔗
|
|
dashcloud has quit IRC (Read error: Operation timed out) |
|
04:15
🔗
|
|
dashcloud has joined #archiveteam-bs |
|
05:14
🔗
|
|
vitzli has joined #archiveteam-bs |
|
05:21
🔗
|
|
Stiletto has joined #archiveteam-bs |
|
05:36
🔗
|
|
Sk1d has quit IRC (Ping timeout: 194 seconds) |
|
05:42
🔗
|
|
Sk1d has joined #archiveteam-bs |
|
06:37
🔗
|
|
Start_ has quit IRC (Quit: Disconnected.) |
|
06:37
🔗
|
|
Start has joined #archiveteam-bs |
|
06:43
🔗
|
|
nicolas17 has quit IRC (Quit: nuff 4 2day) |
|
07:07
🔗
|
|
jspiros has quit IRC (Read error: Operation timed out) |
|
07:08
🔗
|
|
jspiros has joined #archiveteam-bs |
|
07:21
🔗
|
|
vitzli has quit IRC (Quit: Leaving) |
|
07:25
🔗
|
|
vitzli has joined #archiveteam-bs |
|
07:42
🔗
|
|
ravetcofx has quit IRC (Read error: Operation timed out) |
|
08:11
🔗
|
|
krazedkat has quit IRC (Ping timeout: 244 seconds) |
|
08:17
🔗
|
|
SadDM has quit IRC (Read error: Operation timed out) |
|
08:17
🔗
|
|
SadDM has joined #archiveteam-bs |
|
08:17
🔗
|
|
swebb sets mode: +o SadDM |
|
08:32
🔗
|
|
SadDM has quit IRC (Read error: Operation timed out) |
|
08:40
🔗
|
|
SadDM has joined #archiveteam-bs |
|
08:40
🔗
|
|
swebb sets mode: +o SadDM |
|
08:52
🔗
|
|
GE has joined #archiveteam-bs |
|
09:02
🔗
|
|
SadDM has quit IRC (Read error: Operation timed out) |
|
09:05
🔗
|
|
SadDM has joined #archiveteam-bs |
|
09:05
🔗
|
|
swebb sets mode: +o SadDM |
|
09:32
🔗
|
|
SadDM has quit IRC (Read error: Operation timed out) |
|
09:35
🔗
|
|
SadDM has joined #archiveteam-bs |
|
09:35
🔗
|
|
swebb sets mode: +o SadDM |
|
09:41
🔗
|
|
SadDM has quit IRC (Read error: Operation timed out) |
|
09:47
🔗
|
|
SadDM has joined #archiveteam-bs |
|
09:47
🔗
|
|
swebb sets mode: +o SadDM |
|
09:49
🔗
|
|
dashcloud has quit IRC (Read error: Operation timed out) |
|
09:53
🔗
|
|
dashcloud has joined #archiveteam-bs |
|
10:00
🔗
|
|
SadDM has quit IRC (Read error: Operation timed out) |
|
10:02
🔗
|
|
SadDM has joined #archiveteam-bs |
|
10:02
🔗
|
|
swebb sets mode: +o SadDM |
|
10:07
🔗
|
|
SadDM has quit IRC (Read error: Operation timed out) |
|
10:08
🔗
|
|
SadDM has joined #archiveteam-bs |
|
10:08
🔗
|
|
swebb sets mode: +o SadDM |
|
10:11
🔗
|
|
BlueMaxim has quit IRC (Quit: Leaving) |
|
11:06
🔗
|
|
krazedkat has joined #archiveteam-bs |
|
11:47
🔗
|
|
BartoCH has quit IRC (Ping timeout: 260 seconds) |
|
11:50
🔗
|
|
BartoCH has joined #archiveteam-bs |
|
12:04
🔗
|
|
GE has quit IRC (Remote host closed the connection) |
|
12:45
🔗
|
|
Asparagir has quit IRC (Read error: Operation timed out) |
|
12:56
🔗
|
|
Asparagir has joined #archiveteam-bs |
|
13:31
🔗
|
|
GE has joined #archiveteam-bs |
|
13:51
🔗
|
|
fie has quit IRC (Ping timeout: 506 seconds) |
|
14:15
🔗
|
|
i336_ has quit IRC (Ping timeout: 260 seconds) |
|
14:53
🔗
|
|
vitzli has quit IRC (Quit: Leaving) |
|
14:59
🔗
|
|
dashcloud has quit IRC (Read error: Operation timed out) |
|
15:04
🔗
|
|
dashcloud has joined #archiveteam-bs |
|
15:42
🔗
|
|
fie has joined #archiveteam-bs |
|
15:57
🔗
|
|
fie has quit IRC (Read error: Operation timed out) |
|
16:51
🔗
|
godane |
i'm grabbing descriptions of the Rush Limbaugh show going back 6 years |
|
16:55
🔗
|
|
dashcloud has quit IRC (Read error: Operation timed out) |
|
16:59
🔗
|
|
dashcloud has joined #archiveteam-bs |
|
17:07
🔗
|
|
HCross has quit IRC (Read error: Operation timed out) |
|
17:13
🔗
|
|
HarryCros has joined #archiveteam-bs |
|
17:21
🔗
|
|
ndiddy has joined #archiveteam-bs |
|
17:43
🔗
|
|
fie has joined #archiveteam-bs |
|
17:47
🔗
|
|
nicolas17 has joined #archiveteam-bs |
|
18:31
🔗
|
PurpleSym |
SketchCow: I saw you moved my Yahoo Groups crawl into the archiveteam/web collection. Given the number of items created so far, would it make sense to create a separate collection just for this data? With proper permissions I could organize new uploads into that new collection myself. |
|
18:52
🔗
|
|
Rye has quit IRC (Quit: ZNC - http://znc.in) |
|
18:55
🔗
|
|
Rye has joined #archiveteam-bs |
|
18:55
🔗
|
|
Rye has quit IRC (Remote host closed the connection) |
|
18:57
🔗
|
|
Rye has joined #archiveteam-bs |
|
19:04
🔗
|
|
Rye has quit IRC (Quit: ZNC - http://znc.in) |
|
19:08
🔗
|
|
Rye has joined #archiveteam-bs |
|
19:26
🔗
|
|
brayden has quit IRC (Ping timeout: 633 seconds) |
|
20:47
🔗
|
|
whopper has quit IRC (hub.se irc.efnet.nl) |
|
20:47
🔗
|
|
zerkalo has quit IRC (hub.se irc.efnet.nl) |
|
20:47
🔗
|
|
wacky has quit IRC (hub.se irc.efnet.nl) |
|
20:47
🔗
|
|
luckcolor has quit IRC (hub.se irc.efnet.nl) |
|
20:47
🔗
|
|
w0pr has joined #archiveteam-bs |
|
20:47
🔗
|
|
zerkalo_ has joined #archiveteam-bs |
|
20:52
🔗
|
|
wacky_ has joined #archiveteam-bs |
|
21:03
🔗
|
|
luckcolor has joined #archiveteam-bs |
|
21:16
🔗
|
|
RichardG_ has joined #archiveteam-bs |
|
21:19
🔗
|
|
RichardG has quit IRC (Read error: Operation timed out) |
|
22:05
🔗
|
|
t2t2 has quit IRC (Ping timeout: 260 seconds) |
|
22:32
🔗
|
|
BlueMaxim has joined #archiveteam-bs |
|
22:43
🔗
|
|
yipdw has quit IRC (Quit: yipdw) |
|
22:44
🔗
|
|
yipdw has joined #archiveteam-bs |
|
22:44
🔗
|
|
Frogging sets mode: +o yipdw |
|
22:50
🔗
|
|
GE has quit IRC (Quit: zzz) |
|
22:58
🔗
|
|
t2t2 has joined #archiveteam-bs |
|
23:51
🔗
|
|
i336_ has joined #archiveteam-bs |