| Time |
Nickname |
Message |
|
00:10
🔗
|
|
systwi has joined #urlteam |
|
00:58
🔗
|
|
asdf0101 has quit IRC (The Lounge - https://thelounge.chat) |
|
01:06
🔗
|
|
asdf0101 has joined #urlteam |
|
01:42
🔗
|
|
kiska1 has quit IRC (Read error: Operation timed out) |
|
02:03
🔗
|
|
kiska1 has joined #urlteam |
|
02:03
🔗
|
|
Fusl sets mode: +o kiska1 |
|
03:30
🔗
|
|
odemg has quit IRC (Ping timeout: 265 seconds) |
|
16:54
🔗
|
|
Dj-Wawa has joined #urlteam |
|
17:04
🔗
|
|
bajoodle has joined #urlteam |
|
17:04
🔗
|
bajoodle |
Hi guys :) |
|
17:04
🔗
|
bajoodle |
When running the client-grab (https://github.com/ArchiveTeam/terroroftinytown-client-grab) is there any max on the concurrency? Is it 6 as it is in the warrior? |
|
17:05
🔗
|
bajoodle |
concurrent items* |
|
17:17
🔗
|
JAA |
bajoodle: seesaw enforces a maximum of 20 IIRC, after that you need to run the pipeline multiple times. However, URLTeam further enforces one job per shortener and external IP due to rate limiting and bans by the shorteners we're scraping, so going to high numbers doesn't really do anything. |
|
17:17
🔗
|
JAA |
(On a single machine, that is.) |
|
17:18
🔗
|
bajoodle |
JAA: Alright, so I might as well keep it at 6 when running on a single machine? |
|
17:18
🔗
|
JAA |
Yeah, probably. We don't have many shorteners active at the moment, and there are more workers than shorteners anyway. |
|
17:19
🔗
|
bajoodle |
Alright, I'll keep it at 6 then, thanks! :) |
|
17:19
🔗
|
JAA |
Also, if you have multiple IPs on one machine, you'll have to run one pipeline for each IP, and a high concurrency on each pipeline would again be useless. |
|
17:21
🔗
|
bajoodle |
Was just wondering as I could see that those in the "top" of the leaderboard seemed to be scanning a whole-lot more than I do, I suppose they're just running on multiple machines then |
|
17:22
🔗
|
|
Kagee has quit IRC (Quit: WeeChat 2.4) |
|
17:23
🔗
|
JAA |
Probably, yes. |
|
17:36
🔗
|
|
bajoodle has quit IRC (Quit: Page closed) |
|
18:19
🔗
|
|
Kagee has joined #urlteam |
|
22:28
🔗
|
|
Jens has quit IRC (Remote host closed the connection) |
|
22:28
🔗
|
|
Jens has joined #urlteam |