#newsgrabber 2017-11-17,Fri

Logs of this channel are not protected. You can protect them by a password.

↑back Search ←Prev date Next date→ Show only urls(Click on time to select a line by its url)


WhoWhatWhen
***blitzed has quit IRC (Remote host closed the connection) [08:46]
............................................................................... (idle for 6h31mn)
jrwrI've gotten two more seedboxes for newsgrabber
spreading them out a little
(to be used as warriors)
[15:17]
............. (idle for 1h3mn)
***blitzed has joined #newsgrabber [16:20]
AoedeINFO Fetched ‘https://www.nikkei.com/article/DGXMZO23205850X01C17A1TJ1000/nkds.slider.min.js/nkds.classchange.min.js/nkds.loadswf.viewer.min.js/’: 200 OK. Length: 187519 [text/html;charset=utf-8].
wpull loops like this on nikkey.com urls, oddly enough this only happens on one machine
after few GBs it completes
[16:23]
.......................................... (idle for 3h29mn)
***blitzed has quit IRC (Quit: Leaving) [19:53]
jrwrso arkiver we are getting some nice traffic stats now [19:59]
....... (idle for 34mn)
JensRexPlease slow down grabbing. There's no way I'll make my way to the leaderboard at this rate! [20:33]
jrwrjrwr spins up 40 more VPS [20:38]
Kazright
i think it's time to give Oles some more money
[20:39]
jrwrI wan that req/s to hit 400 by end of day dammit
Kaz THROW MORE MONEY AT IT
[20:40]
KazMORE [20:40]
jrwrPOOOOWWWWEEEERRRRRR [20:40]
Kaznow we play the game of making sure I use the right card
so I don't wait all weekend for approval
[20:41]
JensRexI've made a huge mistake. [20:41]
Kaz' We check your payment manually '
ffs
[20:42]
jrwrKAz
Get some feral slots
[20:42]
Kazhow mad are they going to get when we start capping the CPU on all their hosts? [20:43]
jrwrruns the pipeline fine, just make sure to do install --user and download youtube-dl into the root of pipeline.py
HEHEHE
Jokes on you, I have 5 already running across 5 hosts
concur of 10
[20:43]
JensRexI only run concurrent 1. Had weird problems with large concurrent. [20:44]
Kaz:O OVH approved me
somehow
JensRex: lots of pipelines with low concurrent is better than 1 pipeline with high concurrent
[20:44]
JensRexI'm only running 2 pipelines. One on my DO instance, and one in a VM at home.
What concurrency per pipeline do you recommend?
I can't run >1 on DO, because occasionally I get huge jobs, and my DO node is low on disk space.
[20:45]
Kaz5 per pipeline is my go-to [20:48]
jrwrits a 1TB disk with lots of ram/cpu so
feral slots FTW
HCross2: Whats the limit of BW on the CDN before we start to worry? 1-2TB?
[20:52]
HCross2Not sure [20:56]
jrwrOk
since req/s doesn't matter
(im getting like 20/s on the backend)
[20:56]
***figpucker has joined #newsgrabber [21:00]
IglooI'm punching ~120 concurrency over all my hosts
Averaging about 500Mbps download across that lot
[21:00]
Kazhuh, SYS comes with autoconfigured ipv6 now. sweet [21:02]
IglooHave you gone for a UK server?
I have some in JOE's DC in the US, I'm quite impressed
(I got one of the last ones with SSD)
[21:03]
Kaznah, UK pricing is too much [21:08]
Iglooheh, Kaz I just got to the end of migrating of one of my web servers (physical) which has 3 years uptime (installed it 5 years ago). It's had "root" sshable evidently since forever. [21:09]
Kazouch
hmm, i've got a very sad pipeline
https://www.irccloud.com/pastebin/15QPp8GW/
[21:09]
IglooThat's brokey [21:09]
Kazit's not happy, that's for sure
which version of wpull is everyone using these days?
[21:11]
jrwrthe one off the github
Kaz: thats a DNS error
[21:16]
Kazworking perfectly after switching to google dns
cheers jrwr
[21:18]
hanging for a long time on Starting WgetDownload, hmm
it eventually gets going, but locks all threads for that period of time (probs 1-2 minutes)
seesaw is fun
[21:26]
.......... (idle for 46mn)
nvm, we'll take all of these issues and put them in the pile with the rest of the things OVH can't do properly [22:12]
JAA"I'll put this over here with the rest of the fire".jpg [22:13]
KazI get the feeling that IPv6 was never passing traffic
and my issue was just a timeout before falling back to ipv4
but IPv6 falls under 'things I'm not debugging at this time'
https://s.kurt.gg/19dWezTk.png
i understand nothing
[22:14]
JAAI see a suspicious "Modified" in the top right corner of nano. [22:19]
Kazignore that, that's just cause I dropped some newlines to make it look a bit cleaner
this is post-reboot, too
[22:20]
JAAHuh [22:21]
Kaz*nothing* makes sense
no /etc/network/interfaces file
and /etc/network/interfaces.d is empty
[22:21]
JAAAsk Ubuntu seems to suggest that you need a .disable_ipv6 directive for each device, i.e. that "all" is not sufficient: https://askubuntu.com/a/672302 [22:23]
Kazhang on
i just realised
..systemd
Debian isn't on upstart any more, now everything makes total sense
[22:23]
JAAHm? [22:27]
Kaztwo things
first of all, you're right with the .disable_ipv6 bit, cheers
second, I have no /etc/network/interfaces file because that's from upstart
https://s.kurt.gg/19fOXmYw.png
[22:27]
JAAAh
What Debian version are you on?
[22:29]
Kaz9 [22:29]
JAAHuh
The OVH image?
[22:29]
Kazyup [22:30]
JAAInteresting [22:30]
Kaz9.2, stock kernel rather than the grsec kernel [22:30]
JAASo I'm using stretch as well, but installed from the official netinst images. I have /etc/network/interfaces and am using systemd.
Also, /etc/network/interfaces doesn't really have anything to do with upstart as far as I know.
[22:33]
Kazwell, I'm happy to resign to 'I haveno idea what I'm doing'
output of uname -a?
I'm interested
[22:34]
JAAHaha, same. Network configuration is hell.
4.9.0-4-amd64 #1 SMP Debian 4.9.51-1 (2017-09-28) x86_64 GNU/Linux
[22:34]
Kazexactly the same..
you got anything under /etc/systemd/network ?
[22:35]
JAANope, empty directory. [22:35]
Kazinteresting
i guess OVH installer pushes the network config through systemd directly, rather than whatever /etc/network/interfaces falls under
[22:36]
JAAThat'd be ifupdown.
... which includes /lib/systemd/system/ifup@.service and /lib/systemd/system/networking.service as well as udev rules.
[22:37]
Kazwhat the living..
networking is magic
[22:38]
JAAHehe, yep.
IPv6 is broken on one of my machines. Exactly the same configuration, packages, and everything as another machine where it's working (well, except for the actual IP values, broadcast addresses, etc.).
[22:38]
Kazin ovh? [22:39]
JAANo
netcup, a German provider.
[22:39]
Kazah
I'm 99% sure I wasn't passing any traffic over ipv6
[22:40]
JAAI'm pretty sure at this point that it's an issue on their side. Can't be bothered to fix it though since I'm not using IPv6 anyway. (It's an ArchiveBot pipeline.) [22:40]
HCross2I've moved a lot of my personal stuff from OVH now [22:42]
JAAWhere to, if you don't mind me asking? [22:42]
HCross2Clouvider
Small little UK setup, but is significantly better
He was happy to faff around with a custom network setup for me
[22:42]
KazI was going to colo there.. but £80/mo is steep :( [22:44]
HCross2Eh, it's worth it [22:45]
JAALooks decent [22:45]
HCross2It's where my Archivebot pipeline is etc [22:45]
JAAWhat's the situation with VAT in the UK for non-EU residents?
Same as EU?
[22:45]
HCross2His Enfield DC isn't too far for me
Yep
You don't pay
[22:45]
JAAUh, yeah, about that... [22:46]
HCross2It'll remove vat when it realises you aren't residents [22:46]
JAAWith German providers, I do have to pay VAT. :-|
I think it's something like "Germany is the country of fulfillment of the contract, so you have to pay VAT". Interestingly enough though, Hetzner does not charge VAT while OVH does (for servers in France).
Anyway, thanks, I'll keep that in mind.
[22:46]
.......... (idle for 45mn)
IglooWhat's the network speed like HCross2 ? [23:33]

↑back Search ←Prev date Next date→ Show only urls(Click on time to select a line by its url)