[00:46] *** arkhive has joined #archiveteam-ot [00:47] yo. It's infuriating talking to Comcast on the phone. [00:49] I have 1Gb/s down 35Mb/s up and get capped at 1024GB/mo. It costs $50 USD more a month, so $150 USD/mo for that speed without a cap. They let you exceed your cap without charging you the first two times... Then it is $10 USD more every 50GB you go over. :( errrr [00:56] *** godane has quit IRC (Quit: Leaving.) [01:10] I will do phone calls for you😂 [01:11] i worked 7 years in a call center, they usually hire warm bodies who know nothing, and from time to time they find people who know something about the calls they are taking, lol [01:12] he tried to say we cap so our servers aren't overwhelmed with traffic [01:12] i can however get anything i want from any telco agent, helps when you know the screens they are on ;) [01:12] it always comes down to $$, pay up, or deal, thats not comcasts problem, but yours lol [01:22] *** BlueMax has joined #archiveteam-ot [01:41] *** Stilett0 has quit IRC () [01:41] *** godane has joined #archiveteam-ot [01:42] *** Stiletto has joined #archiveteam-ot [01:46] *** godane has quit IRC (Client Quit) [01:53] *** godane has joined #archiveteam-ot [01:53] *** svchfoo1 sets mode: +o godane [02:05] *** arkhive has quit IRC () [02:27] I tested the latest snscrape on python 3.6 and 3.7 and I'm pretty sure there's something wrong with how it fetches the continuation for a twitter from: search [02:29] https://twitter.com/search?q=from%3Asiatechhq&src=typd&f=realtime for example returns a lot of results while snscrape returns just 24 [02:34] haha JAA it works after I replace the User-Agent [02:34] headers = {'User-Agent': 'Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/70.0.3538.45 Safari/537.36'} [02:34] verified several times [02:45] now that it's working I'm going to have a truly excessive number of tweets to archive [02:46] Bahahahahahahaha [02:48] *** chfoo has quit IRC (Read error: Operation timed out) [02:48] *** jrwr has quit IRC (Read error: Operation timed out) [02:56] *** chfoo has joined #archiveteam-ot [02:56] *** jrwr has joined #archiveteam-ot [03:29] *** odemg has quit IRC (Ping timeout: 260 seconds) [03:42] *** odemg has joined #archiveteam-ot [04:14] *** Stilett0 has joined #archiveteam-ot [04:16] *** Stiletto has quit IRC (Ping timeout: 268 seconds) [04:17] *** Stiletto has joined #archiveteam-ot [04:20] *** Stilett0 has quit IRC (Read error: Operation timed out) [04:45] *** ivan has quit IRC (Read error: Operation timed out) [04:45] *** JAA has quit IRC (Read error: Operation timed out) [04:46] *** ivan has joined #archiveteam-ot [04:46] *** chfoo has quit IRC (Read error: Operation timed out) [04:46] *** svchfoo1 sets mode: +o ivan [04:46] *** JAA has joined #archiveteam-ot [04:47] *** chfoo has joined #archiveteam-ot [04:47] *** svchfoo3 sets mode: +o JAA [04:47] *** bakJAA sets mode: +o JAA [04:50] *** wp494 has quit IRC (Read error: Operation timed out) [04:50] *** wp494 has joined #archiveteam-ot [04:50] *** svchfoo3 sets mode: +o wp494 [07:03] *** djsundog has quit IRC (Read error: Operation timed out) [07:03] *** djsundog has joined #archiveteam-ot [07:23] *** schbirid has joined #archiveteam-ot [07:41] *** zino has quit IRC (Quit: Leaving) [07:41] *** zino has joined #archiveteam-ot [10:41] *** Stiletto has quit IRC (Ping timeout: 268 seconds) [11:09] *** wp494 has quit IRC (Ping timeout: 252 seconds) [11:09] *** wp494 has joined #archiveteam-ot [11:10] *** svchfoo1 sets mode: +o wp494 [11:10] ivan: Really weird that it doesn't work for you with the 39 UA. I should refactor that UA handling anyway though. [11:22] *** icedice has joined #archiveteam-ot [11:39] arkhive vectr0n it looks like comcast has imported old Australian data caps xD [11:40] JAA: only that exact UA is banned, I changed a digit in the minor version and it worked [11:40] JAA: maybe a frontend firewall that is classified the UA as 100% likely bot [11:40] s/is// [11:41] maybe fake a Chrome minor version? :-) [11:41] Alternatively you can use an actual version of Chrome! [11:41] Like "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/71.0.3569.0 Safari/537.36" [11:42] yes but then you have to update it :-) [11:42] ivan: But it worked fine for me. So maybe it's a ban specific to that UA and your IP(s)? [11:42] someone should make a github bot for projects that need to lie about their UA [11:43] I mean you only need to update it occasionally, once every 12 weeks should be sufficient [11:52] JAA: yeah, it looks like it's keyed on (ip, ua) or similar [11:52] all of my IPs just happened to get into the shitlist [11:53] maybe a time.sleep would help avoid that [11:53] I've only been doing one snscrape at a time per IP [11:58] Yeah, maybe. I haven't yet figured out how to handle rate limiting in snscrape. [11:58] It's also weird that you got 6 results. The initial page should contain 10, I think... [12:51] Oh great, all Travis links on the wpull repo are broken now due to the repo move... [13:15] *** VerifiedJ has joined #archiveteam-ot [13:51] *** kiska1 has quit IRC (Read error: Operation timed out) [13:52] *** kiska has quit IRC (Quit: Ping timeout (120 seconds)) [13:52] *** Flashfire has quit IRC (Quit: Ping timeout (120 seconds)) [13:52] *** w0rmhole has quit IRC (Quit: Ping timeout (120 seconds)) [14:54] *** BlueMax has quit IRC (Quit: Leaving) [16:47] *** Stiletto has joined #archiveteam-ot [16:56] *** Mateon1 has quit IRC (Ping timeout: 360 seconds) [16:56] *** Mateon1 has joined #archiveteam-ot [17:51] *** faolingfa has joined #archiveteam-ot [18:43] *** VerifiedJ has quit IRC (Read error: Operation timed out) [18:54] *** schbirid has quit IRC (Ping timeout: 1212 seconds) [19:09] *** kiska1 has joined #archiveteam-ot [19:09] *** w0rmhole has joined #archiveteam-ot [19:09] *** kiska has joined #archiveteam-ot [19:09] *** Flashfire has joined #archiveteam-ot [19:40] *** Stilett0 has joined #archiveteam-ot [19:42] *** Stiletto has quit IRC (Read error: Operation timed out) [19:49] *** SimpBrain has quit IRC (Ping timeout: 252 seconds) [20:36] https://twitter.com/nickm_tor/status/860234274842324993 [21:04] moufu: Here's why you shouldn't use my code at the moment: https://travis-ci.org/ArchiveTeam/wpull/jobs/439753220 :-) [21:11] Huh, transfer.sh is down. [21:34] *** Stilett0 has quit IRC (Ping timeout: 268 seconds) [21:34] *** Stiletto has joined #archiveteam-ot [21:51] *** kiskabak has quit IRC (Ping timeout: 268 seconds) [21:53] *** w0rmybak has quit IRC (Ping timeout: 268 seconds) [21:56] *** Albardin has quit IRC (Ping timeout: 600 seconds) [22:14] *** Albardin has joined #archiveteam-ot [22:14] *** kiskabak has joined #archiveteam-ot [22:14] *** w0rmybak has joined #archiveteam-ot [23:07] *** SketchCow has joined #archiveteam-ot [23:27] *** BlueMax has joined #archiveteam-ot