#urlteam 2019-06-11,Tue

↑back Search

Time Nickname Message
00:10 🔗 systwi has joined #urlteam
00:58 🔗 asdf0101 has quit IRC (The Lounge - https://thelounge.chat)
01:06 🔗 asdf0101 has joined #urlteam
01:42 🔗 kiska1 has quit IRC (Read error: Operation timed out)
02:03 🔗 kiska1 has joined #urlteam
02:03 🔗 Fusl sets mode: +o kiska1
03:30 🔗 odemg has quit IRC (Ping timeout: 265 seconds)
16:54 🔗 Dj-Wawa has joined #urlteam
17:04 🔗 bajoodle has joined #urlteam
17:04 🔗 bajoodle Hi guys :)
17:04 🔗 bajoodle When running the client-grab (https://github.com/ArchiveTeam/terroroftinytown-client-grab) is there any max on the concurrency? Is it 6 as it is in the warrior?
17:05 🔗 bajoodle concurrent items*
17:17 🔗 JAA bajoodle: seesaw enforces a maximum of 20 IIRC, after that you need to run the pipeline multiple times. However, URLTeam further enforces one job per shortener and external IP due to rate limiting and bans by the shorteners we're scraping, so going to high numbers doesn't really do anything.
17:17 🔗 JAA (On a single machine, that is.)
17:18 🔗 bajoodle JAA: Alright, so I might as well keep it at 6 when running on a single machine?
17:18 🔗 JAA Yeah, probably. We don't have many shorteners active at the moment, and there are more workers than shorteners anyway.
17:19 🔗 bajoodle Alright, I'll keep it at 6 then, thanks! :)
17:19 🔗 JAA Also, if you have multiple IPs on one machine, you'll have to run one pipeline for each IP, and a high concurrency on each pipeline would again be useless.
17:21 🔗 bajoodle Was just wondering as I could see that those in the "top" of the leaderboard seemed to be scanning a whole-lot more than I do, I suppose they're just running on multiple machines then
17:22 🔗 Kagee has quit IRC (Quit: WeeChat 2.4)
17:23 🔗 JAA Probably, yes.
17:36 🔗 bajoodle has quit IRC (Quit: Page closed)
18:19 🔗 Kagee has joined #urlteam
22:28 🔗 Jens has quit IRC (Remote host closed the connection)
22:28 🔗 Jens has joined #urlteam

irclogger-viewer