#newsgrabber 2017-07-15,Sat

Logs of this channel are not protected. You can protect them by a password.

↑back Search ←Prev date Next date→ Show only urls(Click on time to select a line by its url)


WhoWhatWhen
***luckcolor has quit IRC (Read error: Operation timed out)
luckcolor has joined #newsgrabber
[08:16]
.......................... (idle for 2h8mn)
HCross2ive got some utterly huge items on the way in
Vienna discovery soon
[10:25]
.... (idle for 15mn)
or not.. as someone at M247 doesnt understand how ipv6 works [10:42]
........ (idle for 35mn)
jrwr: dedupe is hellishly fast this morning :) [11:17]
trvzI'm running into "Please limit --concurrent to 20 or lower to avoid exhausting resources or triggering bugs."
for more than 20, should I just set up another user?
[11:25]
HCross2just run another instance of the script
use --port xxxx if needs be
I've got a server dpomg 80
doing
[11:26]
trvzyes, I meant another user for another instance [11:27]
HCross2you dont need to use another user [11:27]
trvzcreate a second folder with the warrior code again or just run it in the original folder alongside the other instance? [11:29]
HCross2just run in the original folder [11:29]
trvzgreat, thanks [11:30]
HCross2it handles it as each item sets up a subfolder inside the original folder as its "workspace" [11:30]
trvzsocket.error: [Errno 98] Address already in use
with --address '127.0.0.1'
[11:31]
JAAGood to know that it handles that fine, I usually set up separate directories for the parallel pipelines. [11:31]
trvzaaah
there's a whole section on github for this
sorry
[11:32]
HCross2no problem at all
you just use --port
[11:32]
.... (idle for 19mn)
trvzwhat hardware are you running on with a concurrency of 80? [11:51]
........................ (idle for 1h57mn)
***shinjuken has joined #newsgrabber [13:48]
shinjukenhi
hello
hi
[13:48]
Aoedeo/ [13:49]
shinjukenim new to bitchx [13:50]
***shinjuken has left [13:52]
............ (idle for 59mn)
Kaztrvz: concurrent 5 per instance seems to be best for me [14:51]
HCross2To give some perspective... 30 concurrent is holding a 4 core M247 VPS at a load of 3.6 [15:02]
arkiverIt looks like a discoverer is down
discovered URLs should be like 9000 per 15 minutes
or so
currently it's too low
looks like HCBLRD1 is down
[15:03]
HCross2arkiver: thanks for the heads up. Let me go wrestle with it. Im not too happy with the server that that one is running on - so ill probably move it [15:06]
arkiverok
Didn't that one had problems before?
[15:07]
HCross2yea
it keeps getting killed - its only on 512MB RAM
[15:08]
arkiverah
maybe it can just be given a smaller percentage of services
[15:09]
HCross2its quite nice to have an India based discovery box anyway [15:10]
arkiverAs long as it doesn't block any of the websites
mauybe we can sort websites by country
[15:10]
HCross2shouldnt be too bad as its a server [15:11]
arkiverso a chinese server can discover the chinese websites for example [15:11]
HCross2Kaz: arent you running a discovery behind your BT home line? [15:11]
Kazi am [15:11]
HCross2Hopefully the UK isnt blocking any of our news sites yet [15:11]
Kazyeah, should be okay I think
I've never noticed any issues browsing locally
[15:12]
HCross2although... who knows what may happen [15:12]
arkiverKaz: are 'adult' site blocked by the box?
sites*
UK blocked some of these sites right?
[15:12]
HCross2arkiver: yep [15:12]
Kazehm, most likely [15:13]
arkivergood to keep in mind in case we might add services for 'adult' news sites [15:13]
Kazyeah, worth keeping an eye on [15:13]
HCross2arkiver: im still arguing with my phone provider over archive.org [15:13]
Kaz]it's only DNS level blocking though, isn't it [15:13]
arkiverHCross2: IA knows about it?
they might be able to put some pressure on them too
[15:13]
KazI have DNS overrides for some sites, to get around issues [15:14]
HCross2arkiver: ive been speaking to the London office manager [15:14]
arkiverah yes [15:14]
HCross2I even signed up for a contract and did age validation... still blocked [15:14]
arkiverDNS level block?
jrwr: we had a problem with the scripts adding the archived URLs to redis
it's fixed now
A lot of info is now being added.
[15:15]
HCross2arkiver: I think I just made it work [15:20]
arkivernice!
time for an !rs?
[15:22]
HCross2I meant the IA web blocking - im sorting out HCBLRD1 now [15:22]
arkiverhow did you make it work? [15:23]
HCross2for some reason, they had not taken off the adult web filtering when I took on my contract
taken it off now and it works
they think the IA is a porn filled hellhole
[15:23]
arkiverhaha
well a pretty large part of the wayback machine is
but so is the unblocked internet
uncensored*
[15:24]
HCross2Which of course goes against Mrs May's "protect the children" campaign [15:25]
arkiveryeah... [15:25]
HCross2I think we may need another disco anyway to help with the load
arkiver: I cut HCBLRD1's workload in half
but now HCLUD1 is taking up the strain
[15:27]
arkiveryep, looks good again [15:28]
HCross2running a cat on a 10MB text file over SSH to a server in India was a bad idea [15:29]
jrwrjrwr: what kind of issues
arkiver: here I am talking to my self
[15:34]
arkiverlooks like something could not be loaded first
magically fixed today
[15:37]
jrwrlol
Watch ram on the box if you are ever in it
[15:40]
HCross2jrwr: ive got monthly Azure credits and they do a Redis cache thing - would that be of use? [15:43]
jrwrna
I would say the DB is half full
[15:43]
HCross2but this isnt full whack as weve had a discovery down [15:43]
jrwrtrue
Gotta Think about speed here as well
give it 48 hours let see how it Chooch
[15:44]
HCross2arkiver: how are we doing in terms of capacity - do we need a lot more? [15:52]
jrwrI will say that nginx is getting 80% of the load
With the on disk cache
hows the workers doing
is it really fast?
[15:52]
HCross2jrwr: yep
M247 Frankfurt is 13ms away and its flying past
[15:53]
jrwrI /might/ switch from redis and go back to a RMDBS
Its about 10x slower on the backend tho
[15:54]
HCross2Im writing a lot of revist records - especially around facebook urls [15:54]
jrwrLets give it 48 hours with the new discovery [15:55]
HCross2jrwr: https://gyazo.com/457ee018042132ccd9137ad8ad320f24 [15:56]
jrwrhttps://www.youtube.com/watch?v=atuFSv2bLa8
better yet
https://www.youtube.com/watch?v=dv13gl0a-FA
since its a dedupe process
jrwr is making really bad jokes now
[15:59]
HCross2*badumph tss* [16:00]
jrwr*ZOOM* *ZOOM* *ZOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMM*
Ive never had nginx eat so much CPU in a project before
this is great knowledge as a webdev
mind you we are doing something like 2k/req/s
[16:00]
HCross2jrwr: we should move this to shared hosting :p [16:02]
jrwrI've though about it
flat file the whole damn thing
[16:02]
HCross2and it would be the fastest ever time someones been thrown out of a shared web host [16:03]
jrwrWhen I was doing /r/place videos
I needed mirrors for the video since I was doing the raw video
it was only 50mb mind you
I got kicked of 4 seedboxes
for pulling 40Gbit/s
[16:03]
HCross2I remember our first Online.net server for this project - when it was 1 server doing everything... 100TB a month got me a very stern email asking me for another 100 eur a month or to find another host [16:04]
jrwrIve been kicked off strange places befor
Whoops, pulled 1PB this month
(Rouge Script going nuts)
[16:06]
AoedeO.o [16:06]
jrwrIt was on a feral seedbox
I was eating ALL the BW
[16:07]
Aoedewhat you were DL'ing? [16:07]
jrwr/Things/ [16:08]
Aoedehehe [16:09]
***HarryCros has joined #newsgrabber
HarryCros is now known as HCross
[16:09]
HCrosstest test [16:09]
Aoedeonly thing happen to me was getting my ip banned from some wiki
pong
[16:09]
HCrossgood good, this works
Aoede, Ive had a DC claim that I was DoS'ing while uploading to the IA
[16:10]
Aoedehahaha
that's something
[16:12]
jrwrIve had that before [16:12]
Aoede:D [16:12]
jrwrI was running a mumble server
with 4000 nerds in it
all UDP traffic
one AAAAAHHHHHHHHHHHH Chain == Massive DDoS
[16:12]
HCrossAoede, was wonderful.. 3am call to ask "what on earth are you doing" [16:13]
jrwrFYi
getting 4000 nerds to all AHH at the same time is amazing
[16:17]
HCrosslol ive seen it happen before. https://www.reddit.com/r/sysadmin/comments/4hx9w4/eve_online_guild_moves_voice_chat_to_a_new_host/ [16:20]
jrwrThats MY post :) [16:20]
HCrossohhh loll
I remember the fallout on LET from this
[16:21]
jrwrWell my post in proxy, we are a IT team at TEST [16:21]
HCrossits a small world [16:21]
AoedeDramatic EVE stuff is always fun to watch. [16:24]
jrwrOh god yes
We have better SSO then more Enterprise Settings
most*
EVE API -> self register -> Auto Groups based on in-game roles -> Mumble, Jabber, Forums, Tools
If you ever left, Bam, all access revoked
Also, they would pull all your game details and spy on their users to prevent spies
They would catch 2-3 spies a week!
Mostly since we had spies elsewhere and would cross check IP Addresses and shit
If you took a screenshot of our forums, the post had hidden details in it no matter what you did (even a copy paste) we could extract 15 bits of user data
[16:24]
HCrosshmm - ive got my eye on some of the ArchiveTeam stickers
however shipping to the UK is more than 1 sticker
[16:32]
underscorwow scaleway is pretty awesome [16:46]
HCrosseh, im not too big a fan of them
after they failed to launch an instance, and still billed me 30 eur for it
[16:46]
underscorwhat do you like for similar microvms?
oh wow
[16:46]
HCrossatm im quite liking https://www.vps247.com [16:47]
jrwrSo
The stickers are legit, they even work well on my black coffee mug
and my car
[16:48]
HCrossjrwr, yea - may get a few [16:48]
jrwrIt starts a discussion thats for sure [16:50]
.......................... (idle for 2h8mn)
HCross2arkiver: your opinion on the new Dutch tapping laws? [18:58]
.............. (idle for 1h7mn)
***Administr has joined #newsgrabber
HCross has quit IRC (Ping timeout: 268 seconds)
[20:05]
ErkDog has quit IRC (ny.us.hub hub.efnet.us)
kurt_ has quit IRC (ny.us.hub hub.efnet.us)
Igloo_ has quit IRC (ny.us.hub hub.efnet.us)
Fletcher| has quit IRC (ny.us.hub hub.efnet.us)
Fletcher- has quit IRC (ny.us.hub hub.efnet.us)
underscor has quit IRC (ny.us.hub hub.efnet.us)
joepie91 has quit IRC (ny.us.hub hub.efnet.us)
luckcolor has quit IRC (ny.us.hub hub.efnet.us)
chfoo has quit IRC (ny.us.hub hub.efnet.us)
kyan has quit IRC (ny.us.hub hub.efnet.us)
MrRadar has quit IRC (ny.us.hub hub.efnet.us)
midas1 has quit IRC (ny.us.hub hub.efnet.us)
dxrt has quit IRC (ny.us.hub hub.efnet.us)
arkiver has quit IRC (ny.us.hub hub.efnet.us)
midas has quit IRC (ny.us.hub hub.efnet.us)
lainu has quit IRC (ny.us.hub hub.efnet.us)
SmileyG has quit IRC (ny.us.hub hub.efnet.us)
stns4_ has quit IRC (ny.us.hub hub.efnet.us)
jrwr has quit IRC (ny.us.hub hub.efnet.us)
ivan has quit IRC (ny.us.hub hub.efnet.us)
JAA has quit IRC (ny.us.hub hub.efnet.us)
MrRadar has joined #newsgrabber
luckcolor has joined #newsgrabber
kyan has joined #newsgrabber
chfoo has joined #newsgrabber
ErkDog has joined #newsgrabber
SmileyG has joined #newsgrabber
Fletcher| has joined #newsgrabber
Fletcher- has joined #newsgrabber
kurt_ has joined #newsgrabber
Igloo_ has joined #newsgrabber
stns4_ has joined #newsgrabber
jrwr has joined #newsgrabber
ivan has joined #newsgrabber
underscor has joined #newsgrabber
midas1 has joined #newsgrabber
joepie91 has joined #newsgrabber
JAA has joined #newsgrabber
lainu has joined #newsgrabber
midas has joined #newsgrabber
arkiver has joined #newsgrabber
dxrt has joined #newsgrabber
hub.efnet.us sets mode: +o arkiver
[20:21]
ErkDog has quit IRC (ny.us.hub hub.efnet.us)
kurt_ has quit IRC (ny.us.hub hub.efnet.us)
Igloo_ has quit IRC (ny.us.hub hub.efnet.us)
Fletcher| has quit IRC (ny.us.hub hub.efnet.us)
Fletcher- has quit IRC (ny.us.hub hub.efnet.us)
underscor has quit IRC (ny.us.hub hub.efnet.us)
joepie91 has quit IRC (ny.us.hub hub.efnet.us)
luckcolor has quit IRC (ny.us.hub hub.efnet.us)
chfoo has quit IRC (ny.us.hub hub.efnet.us)
MrRadar has quit IRC (ny.us.hub hub.efnet.us)
kyan has quit IRC (ny.us.hub hub.efnet.us)
midas1 has quit IRC (ny.us.hub hub.efnet.us)
dxrt has quit IRC (ny.us.hub hub.efnet.us)
arkiver has quit IRC (ny.us.hub hub.efnet.us)
midas has quit IRC (ny.us.hub hub.efnet.us)
lainu has quit IRC (ny.us.hub hub.efnet.us)
SmileyG has quit IRC (ny.us.hub hub.efnet.us)
stns4_ has quit IRC (ny.us.hub hub.efnet.us)
jrwr has quit IRC (ny.us.hub hub.efnet.us)
ivan has quit IRC (ny.us.hub hub.efnet.us)
JAA has quit IRC (ny.us.hub hub.efnet.us)
[20:33]
kimmer has joined #newsgrabber
MrRadar has joined #newsgrabber
luckcolor has joined #newsgrabber
kyan has joined #newsgrabber
chfoo has joined #newsgrabber
ErkDog has joined #newsgrabber
SmileyG has joined #newsgrabber
Fletcher| has joined #newsgrabber
Fletcher- has joined #newsgrabber
kurt_ has joined #newsgrabber
Igloo_ has joined #newsgrabber
stns4_ has joined #newsgrabber
jrwr has joined #newsgrabber
ivan has joined #newsgrabber
underscor has joined #newsgrabber
midas1 has joined #newsgrabber
joepie91 has joined #newsgrabber
JAA has joined #newsgrabber
lainu has joined #newsgrabber
midas has joined #newsgrabber
arkiver has joined #newsgrabber
dxrt has joined #newsgrabber
ircd.choopa.net sets mode: +o arkiver
[20:41]
.............. (idle for 1h6mn)
trvzso from now on, I can let the warrior run indefinitely? [21:47]
jrwrYEs [21:49]
....................... (idle for 1h54mn)
***kyan has quit IRC (Remote host closed the connection) [23:43]

↑back Search ←Prev date Next date→ Show only urls(Click on time to select a line by its url)