| Time |
Nickname |
Message |
|
01:32
🔗
|
|
VADemon has joined #urlteam |
|
04:08
🔗
|
|
arkiver has quit IRC (Read error: Operation timed out) |
|
04:09
🔗
|
|
Wingy has quit IRC (Read error: Operation timed out) |
|
04:13
🔗
|
|
klg has quit IRC (Read error: Operation timed out) |
|
04:15
🔗
|
|
systwi_ has joined #urlteam |
|
04:16
🔗
|
|
arkiver has joined #urlteam |
|
04:16
🔗
|
|
systwi has quit IRC (Ping timeout: 622 seconds) |
|
04:17
🔗
|
|
svchfoo1 sets mode: +o arkiver |
|
04:17
🔗
|
|
Wingy has joined #urlteam |
|
04:17
🔗
|
|
svchfoo3 sets mode: +o arkiver |
|
04:19
🔗
|
|
ranma has quit IRC (Read error: Operation timed out) |
|
04:20
🔗
|
|
ranma has joined #urlteam |
|
04:21
🔗
|
|
klg has joined #urlteam |
|
04:57
🔗
|
|
kiska18 has quit IRC (Read error: Operation timed out) |
|
04:59
🔗
|
|
kiska18 has joined #urlteam |
|
04:59
🔗
|
|
svchfoo1 sets mode: +o kiska18 |
|
04:59
🔗
|
|
svchfoo3 sets mode: +o kiska18 |
|
09:03
🔗
|
|
DLoader has joined #urlteam |
|
10:09
🔗
|
|
tech234a has quit IRC (Quit: Connection closed for inactivity) |
|
11:53
🔗
|
|
Kagee has quit IRC (Ping timeout: 252 seconds) |
|
12:06
🔗
|
|
Kagee has joined #urlteam |
|
14:24
🔗
|
|
VADemon has quit IRC (Read error: Connection reset by peer) |
|
14:25
🔗
|
|
VADemon has joined #urlteam |
|
16:23
🔗
|
Wingy |
Is it normal to get 403 whenever I try to use bitly? I have 14 million scanned urls |
|
16:23
🔗
|
Wingy |
From my home ip |
|
16:24
🔗
|
Wingy |
I've been running 3 instances on my home ip for a few months now |
|
16:24
🔗
|
Wingy |
Each with 6 threads |
|
16:56
🔗
|
|
systwi has joined #urlteam |
|
17:03
🔗
|
|
systwi_ has quit IRC (Ping timeout: 622 seconds) |
|
17:03
🔗
|
JAA |
Yeah, I think we need to increase the sleep between requests to bit.ly. It bans quite quickly. |
|
20:56
🔗
|
abstract |
was the 6-threads-per-instance thing not picked to prevent this? |
|
20:57
🔗
|
abstract |
seems strange to say "im running 3x the limit recommended to avoid catching blocks and now im getting blocked" |
|
21:51
🔗
|
JAA |
That's something separate. The URLTeam tracker will anyway limit you to one item per shortener and IP address. |
|
22:10
🔗
|
Wingy |
Oh so I'm only supposed to run one instance? |
|
22:11
🔗
|
Wingy |
The script allows 20 threads so I thought that threads wouldn't be the limit. |
|
22:23
🔗
|
JAA |
Well... |
|
22:23
🔗
|
JAA |
You can run pretty much any concurrency against this project without issues, as long as you don't murder the tracker. It's pretty useless to do so though. |
|
22:23
🔗
|
JAA |
Problems happen when you run that much concurrency against the default project rather than explicitly against URLTeam. |
|
22:24
🔗
|
JAA |
Because when we change the default project to something else, you suddenly do get that many tasks, and then quickly (a) you get banned, (b) your machine runs out of RAM (c) or disk, etc. |
|
22:27
🔗
|
Wingy |
I use docker-compose to scale specific projects :) |
|
22:28
🔗
|
Wingy |
I don't have anything set to AT Choice |
|
22:28
🔗
|
Wingy |
I might spin an AT Choice container up actually |
|
22:29
🔗
|
JAA |
That's fine. |
|
22:29
🔗
|
JAA |
No point in running too much on URLTeam though due to those per-IP limits. |
|
22:29
🔗
|
JAA |
We really need to add more shorteners. |