| Time |
Nickname |
Message |
|
05:26
🔗
|
|
n9nes has quit IRC (Quit: ZNC 1.7.5 - https://znc.in) |
|
05:27
🔗
|
|
n9nes has joined #urlteam |
|
07:21
🔗
|
|
n9nes has quit IRC (Quit: ZNC 1.7.5 - https://znc.in) |
|
07:21
🔗
|
|
n9nes has joined #urlteam |
|
07:59
🔗
|
|
britmob has quit IRC (Read error: Connection reset by peer) |
|
08:03
🔗
|
|
britmob has joined #urlteam |
|
09:04
🔗
|
|
Datechnom has quit IRC (Read error: Operation timed out) |
|
09:12
🔗
|
|
RBecker has quit IRC (Read error: Operation timed out) |
|
09:12
🔗
|
|
RBecker has joined #urlteam |
|
09:18
🔗
|
|
Ryz has quit IRC (Ping timeout: 496 seconds) |
|
10:11
🔗
|
|
Ryz has joined #urlteam |
|
10:22
🔗
|
|
n9nes has quit IRC (Quit: ZNC 1.7.5 - https://znc.in) |
|
10:22
🔗
|
|
n9nes has joined #urlteam |
|
11:33
🔗
|
|
Ryz has quit IRC (Remote host closed the connection) |
|
11:33
🔗
|
|
kiska1825 has quit IRC (Remote host closed the connection) |
|
11:34
🔗
|
|
Ryz has joined #urlteam |
|
11:34
🔗
|
|
kiska1825 has joined #urlteam |
|
12:18
🔗
|
|
Datechnom has joined #urlteam |
|
13:51
🔗
|
|
seatsea has quit IRC (Quit: The Lounge - https://thelounge.github.io) |
|
13:52
🔗
|
|
seatsea has joined #urlteam |
|
14:07
🔗
|
n9nes |
is there any reason why we cant set max tasks higher than 6? |
|
14:20
🔗
|
phuzion_ |
Mostly to reduce the likelihood of your IP getting banned by various shorteners. |
|
14:20
🔗
|
|
phuzion_ is now known as phuzion |
|
14:24
🔗
|
n9nes |
thanks phuzion |
|
14:24
🔗
|
phuzion |
Yup |
|
14:25
🔗
|
n9nes |
the potential ban will be temporarily right? |
|
14:25
🔗
|
n9nes |
I could try with 5 containers instead of just 1 so :D |
|
14:25
🔗
|
n9nes |
or even more, idk |
|
15:23
🔗
|
phuzion |
n9nes: depends on the shortener, tbh. |
|
15:23
🔗
|
phuzion |
They could ban you for 15 minutes, a few hours, or permanently. |
|
15:30
🔗
|
Craigle |
If I recall the jobs are handed out based on IP. So running a bunch of Warriors will still get you the same amount of jobs. |
|
15:31
🔗
|
Craigle |
Not to mention that getting banned potentially messes up the archiving |
|
17:00
🔗
|
|
systwi_ has joined #urlteam |
|
17:08
🔗
|
|
systwi has quit IRC (Read error: Operation timed out) |
|
17:10
🔗
|
JAA |
n9nes, phuzion: Actually, the main reason for the limit of 6 is that on most projects, more tasks would use too many resources. That's a limit of the warrior VM, not of the underlying scripts or similar. |
|
17:10
🔗
|
phuzion |
Ah gotcha. |
|
17:11
🔗
|
JAA |
For URLTeam specifically, although you could in principle run thousands of concurrency, that doesn't make any sense because the tracker anyway ratelimits you to one item per shortener and IP address. |
|
17:12
🔗
|
JAA |
By the way, if you run the scripts directly instead of the warrior VM, you can go up to 20 concurrent. But again, mostly useless because of the rate limit. |
|
17:13
🔗
|
JAA |
Also, we have way more workers available than items since there are global rate limits per shortener as well so as to not overload the services. That's another reason why most workers will just be stuck with error 404. |
|
17:14
🔗
|
JAA |
In any case, it's not recommended to run multiple pipelines/warriors on the same machine unless you know what you're doing and that your machine can handle the workers being switched to a different, much more resource-intensive project. |
|
17:14
🔗
|
JAA |
And if you do it and get banned from the target website, you'll probably get banned from the tracker as well because that's a waste of time. |
|
18:20
🔗
|
|
britmob has quit IRC (Read error: Connection reset by peer) |
|
18:21
🔗
|
phuzion |
JAA: I suppose we should look into adding more shorteners to the system to give the workers more to work on then, huh? |
|
18:47
🔗
|
|
mtntmnky_ has quit IRC (Remote host closed the connection) |
|
18:48
🔗
|
|
mtntmnky_ has joined #urlteam |
|
18:50
🔗
|
|
britmob has joined #urlteam |
|
18:58
🔗
|
n9nes |
phuzion, JAA thanks for your detailed explanation - so I'll proceeed with one container per project |
|
19:17
🔗
|
JAA |
phuzion: Yes, that's the idea. |
|
19:17
🔗
|
JAA |
n9nes: If you set them to individual projects, not the auto project, then yeah, many things in parallel is fine (as long as the machine can handle it). :-) |
|
19:19
🔗
|
phuzion |
JAA: I'd be more than happy to help, I just need some guidance on prioritizing which projects we should add and which ones we shouldn't. |
|
19:19
🔗
|
phuzion |
I do already have admin creds, fwiw |
|
19:22
🔗
|
JAA |
I'd say anything in danger of shutting down or deleting shortcodes should be prioritised, but other than that, the more, the merrier. |
|
19:23
🔗
|
JAA |
Except bit.ly aliases obviously. |
|
19:36
🔗
|
|
britmob_ has joined #urlteam |
|
19:40
🔗
|
|
britmob has quit IRC (Ping timeout: 276 seconds) |
|
21:21
🔗
|
|
gandalf has quit IRC (Read error: Operation timed out) |
|
21:22
🔗
|
|
gandalf has joined #urlteam |
|
21:36
🔗
|
|
Mayonaise has quit IRC (Read error: Operation timed out) |
|
21:37
🔗
|
|
endrift has quit IRC (Read error: Operation timed out) |
|
21:38
🔗
|
|
paul2520 has quit IRC (Read error: Operation timed out) |
|
21:38
🔗
|
|
jake_test has quit IRC (Read error: Operation timed out) |
|
21:38
🔗
|
|
Mayonaise has joined #urlteam |
|
21:38
🔗
|
|
systwi has joined #urlteam |
|
21:39
🔗
|
|
endrift has joined #urlteam |
|
21:39
🔗
|
|
jake_test has joined #urlteam |
|
21:39
🔗
|
|
paul2520 has joined #urlteam |
|
21:44
🔗
|
|
systwi_ has quit IRC (Read error: Operation timed out) |