#urlteam 2020-04-22,Wed

↑back Search

Time Nickname Message
05:26 🔗 n9nes has quit IRC (Quit: ZNC 1.7.5 - https://znc.in)
05:27 🔗 n9nes has joined #urlteam
07:21 🔗 n9nes has quit IRC (Quit: ZNC 1.7.5 - https://znc.in)
07:21 🔗 n9nes has joined #urlteam
07:59 🔗 britmob has quit IRC (Read error: Connection reset by peer)
08:03 🔗 britmob has joined #urlteam
09:04 🔗 Datechnom has quit IRC (Read error: Operation timed out)
09:12 🔗 RBecker has quit IRC (Read error: Operation timed out)
09:12 🔗 RBecker has joined #urlteam
09:18 🔗 Ryz has quit IRC (Ping timeout: 496 seconds)
10:11 🔗 Ryz has joined #urlteam
10:22 🔗 n9nes has quit IRC (Quit: ZNC 1.7.5 - https://znc.in)
10:22 🔗 n9nes has joined #urlteam
11:33 🔗 Ryz has quit IRC (Remote host closed the connection)
11:33 🔗 kiska1825 has quit IRC (Remote host closed the connection)
11:34 🔗 Ryz has joined #urlteam
11:34 🔗 kiska1825 has joined #urlteam
12:18 🔗 Datechnom has joined #urlteam
13:51 🔗 seatsea has quit IRC (Quit: The Lounge - https://thelounge.github.io)
13:52 🔗 seatsea has joined #urlteam
14:07 🔗 n9nes is there any reason why we cant set max tasks higher than 6?
14:20 🔗 phuzion_ Mostly to reduce the likelihood of your IP getting banned by various shorteners.
14:20 🔗 phuzion_ is now known as phuzion
14:24 🔗 n9nes thanks phuzion
14:24 🔗 phuzion Yup
14:25 🔗 n9nes the potential ban will be temporarily right?
14:25 🔗 n9nes I could try with 5 containers instead of just 1 so :D
14:25 🔗 n9nes or even more, idk
15:23 🔗 phuzion n9nes: depends on the shortener, tbh.
15:23 🔗 phuzion They could ban you for 15 minutes, a few hours, or permanently.
15:30 🔗 Craigle If I recall the jobs are handed out based on IP. So running a bunch of Warriors will still get you the same amount of jobs.
15:31 🔗 Craigle Not to mention that getting banned potentially messes up the archiving
17:00 🔗 systwi_ has joined #urlteam
17:08 🔗 systwi has quit IRC (Read error: Operation timed out)
17:10 🔗 JAA n9nes, phuzion: Actually, the main reason for the limit of 6 is that on most projects, more tasks would use too many resources. That's a limit of the warrior VM, not of the underlying scripts or similar.
17:10 🔗 phuzion Ah gotcha.
17:11 🔗 JAA For URLTeam specifically, although you could in principle run thousands of concurrency, that doesn't make any sense because the tracker anyway ratelimits you to one item per shortener and IP address.
17:12 🔗 JAA By the way, if you run the scripts directly instead of the warrior VM, you can go up to 20 concurrent. But again, mostly useless because of the rate limit.
17:13 🔗 JAA Also, we have way more workers available than items since there are global rate limits per shortener as well so as to not overload the services. That's another reason why most workers will just be stuck with error 404.
17:14 🔗 JAA In any case, it's not recommended to run multiple pipelines/warriors on the same machine unless you know what you're doing and that your machine can handle the workers being switched to a different, much more resource-intensive project.
17:14 🔗 JAA And if you do it and get banned from the target website, you'll probably get banned from the tracker as well because that's a waste of time.
18:20 🔗 britmob has quit IRC (Read error: Connection reset by peer)
18:21 🔗 phuzion JAA: I suppose we should look into adding more shorteners to the system to give the workers more to work on then, huh?
18:47 🔗 mtntmnky_ has quit IRC (Remote host closed the connection)
18:48 🔗 mtntmnky_ has joined #urlteam
18:50 🔗 britmob has joined #urlteam
18:58 🔗 n9nes phuzion, JAA thanks for your detailed explanation - so I'll proceeed with one container per project
19:17 🔗 JAA phuzion: Yes, that's the idea.
19:17 🔗 JAA n9nes: If you set them to individual projects, not the auto project, then yeah, many things in parallel is fine (as long as the machine can handle it). :-)
19:19 🔗 phuzion JAA: I'd be more than happy to help, I just need some guidance on prioritizing which projects we should add and which ones we shouldn't.
19:19 🔗 phuzion I do already have admin creds, fwiw
19:22 🔗 JAA I'd say anything in danger of shutting down or deleting shortcodes should be prioritised, but other than that, the more, the merrier.
19:23 🔗 JAA Except bit.ly aliases obviously.
19:36 🔗 britmob_ has joined #urlteam
19:40 🔗 britmob has quit IRC (Ping timeout: 276 seconds)
21:21 🔗 gandalf has quit IRC (Read error: Operation timed out)
21:22 🔗 gandalf has joined #urlteam
21:36 🔗 Mayonaise has quit IRC (Read error: Operation timed out)
21:37 🔗 endrift has quit IRC (Read error: Operation timed out)
21:38 🔗 paul2520 has quit IRC (Read error: Operation timed out)
21:38 🔗 jake_test has quit IRC (Read error: Operation timed out)
21:38 🔗 Mayonaise has joined #urlteam
21:38 🔗 systwi has joined #urlteam
21:39 🔗 endrift has joined #urlteam
21:39 🔗 jake_test has joined #urlteam
21:39 🔗 paul2520 has joined #urlteam
21:44 🔗 systwi_ has quit IRC (Read error: Operation timed out)

irclogger-viewer