#urlteam 2019-12-06,Fri

↑back Search

Time Nickname Message
01:32 🔗 VADemon has joined #urlteam
04:08 🔗 arkiver has quit IRC (Read error: Operation timed out)
04:09 🔗 Wingy has quit IRC (Read error: Operation timed out)
04:13 🔗 klg has quit IRC (Read error: Operation timed out)
04:15 🔗 systwi_ has joined #urlteam
04:16 🔗 arkiver has joined #urlteam
04:16 🔗 systwi has quit IRC (Ping timeout: 622 seconds)
04:17 🔗 svchfoo1 sets mode: +o arkiver
04:17 🔗 Wingy has joined #urlteam
04:17 🔗 svchfoo3 sets mode: +o arkiver
04:19 🔗 ranma has quit IRC (Read error: Operation timed out)
04:20 🔗 ranma has joined #urlteam
04:21 🔗 klg has joined #urlteam
04:57 🔗 kiska18 has quit IRC (Read error: Operation timed out)
04:59 🔗 kiska18 has joined #urlteam
04:59 🔗 svchfoo1 sets mode: +o kiska18
04:59 🔗 svchfoo3 sets mode: +o kiska18
09:03 🔗 DLoader has joined #urlteam
10:09 🔗 tech234a has quit IRC (Quit: Connection closed for inactivity)
11:53 🔗 Kagee has quit IRC (Ping timeout: 252 seconds)
12:06 🔗 Kagee has joined #urlteam
14:24 🔗 VADemon has quit IRC (Read error: Connection reset by peer)
14:25 🔗 VADemon has joined #urlteam
16:23 🔗 Wingy Is it normal to get 403 whenever I try to use bitly? I have 14 million scanned urls
16:23 🔗 Wingy From my home ip
16:24 🔗 Wingy I've been running 3 instances on my home ip for a few months now
16:24 🔗 Wingy Each with 6 threads
16:56 🔗 systwi has joined #urlteam
17:03 🔗 systwi_ has quit IRC (Ping timeout: 622 seconds)
17:03 🔗 JAA Yeah, I think we need to increase the sleep between requests to bit.ly. It bans quite quickly.
20:56 🔗 abstract was the 6-threads-per-instance thing not picked to prevent this?
20:57 🔗 abstract seems strange to say "im running 3x the limit recommended to avoid catching blocks and now im getting blocked"
21:51 🔗 JAA That's something separate. The URLTeam tracker will anyway limit you to one item per shortener and IP address.
22:10 🔗 Wingy Oh so I'm only supposed to run one instance?
22:11 🔗 Wingy The script allows 20 threads so I thought that threads wouldn't be the limit.
22:23 🔗 JAA Well...
22:23 🔗 JAA You can run pretty much any concurrency against this project without issues, as long as you don't murder the tracker. It's pretty useless to do so though.
22:23 🔗 JAA Problems happen when you run that much concurrency against the default project rather than explicitly against URLTeam.
22:24 🔗 JAA Because when we change the default project to something else, you suddenly do get that many tasks, and then quickly (a) you get banned, (b) your machine runs out of RAM (c) or disk, etc.
22:27 🔗 Wingy I use docker-compose to scale specific projects :)
22:28 🔗 Wingy I don't have anything set to AT Choice
22:28 🔗 Wingy I might spin an AT Choice container up actually
22:29 🔗 JAA That's fine.
22:29 🔗 JAA No point in running too much on URLTeam though due to those per-IP limits.
22:29 🔗 JAA We really need to add more shorteners.

irclogger-viewer