#newsgrabber 2018-01-05,Fri

Logs of this channel are not protected. You can protect them by a password.

↑back Search ←Prev date Next date→ Show only urls(Click on time to select a line by its url)


WhoWhatWhen
***JAA sets mode: -b kimmer*!*@*
Kimmer1 has joined #newsgrabber
[00:09]
..................................................................................................... (idle for 8h23mn)
blitzed has quit IRC (Read error: Connection reset by peer)
blitzed has joined #newsgrabber
[08:34]
..... (idle for 21mn)
blitzed has quit IRC (Read error: Connection reset by peer)
blitzed has joined #newsgrabber
[08:55]
blitzed has quit IRC (Quit: Leaving) [09:03]
.................. (idle for 1h25mn)
jrwrok HCross2 switch the CDN over to http://144.217.66.197:8080
its all ready and going and updating and full
[10:28]
........ (idle for 36mn)
Kimmer1Status on the servers? It looks like I'm having a upload issue. My warriors are at a standstill, where I'm stuck at step 0 of 9 WgetDownload --- and at this step, DeduplicateWarcExtProc. When I look at the leaderboard, it looks to be alive.. I have had this issue for the past week or soo? [11:04]
..... (idle for 20mn)
HCross2jrwr: done [11:24]
.... (idle for 15mn)
***Martle has quit IRC (Read error: Connection reset by peer) [11:39]
............ (idle for 57mn)
medowarHello everyone, I think I´m back. [12:36]
HCross2Hi
jrwr: I briefly broke dedupe this morning.. it's back
[12:45]
IglooThe new dedupe solution is now running [12:54]
...... (idle for 28mn)
Oh, HCross2 / Kaz fyi - New dedupe server is doing 3-3.5 million URLs per 15 mins
The old one was doing ~130,000 from a quick scan
So, Well, Yeah that's a bit faster :o
[13:22]
jrwrYa [13:23]
HCross2My Bunnycdn invoice is feeling the effects too [13:25]
IglooHow much is it? [13:26]
***country51 has joined #newsgrabber
country51 has quit IRC (Remote host closed the connection)
[13:32]
dfoxide has joined #newsgrabber
dfoxide has quit IRC (Remote host closed the connection)
[13:41]
...................... (idle for 1h45mn)
IglooTHink i've found a new issue
Youtube-dl requires openssl > 1.0.2
SInce "somepoint"
[15:26]
jrwroh god
dont get me started on the state of the old ass warriors
[15:36]
IglooI'm planning on fixing it, Just time to do so
Getting them to auto update
But even the scripts are now moaning since i've put the fix in to update youtube-dl on startup :D
[15:37]
.... (idle for 17mn)
Aoedeman this cpu usage
can't run this on my ramnode vps because they'll shut it down lol
[15:55]
Igloowpull isn't very resource friendly [15:56]
JAAIgloo: jrwr made a warrior VM that uses the Docker container. Did you try that? [16:03]
jrwrwarrior3 beta from chfoo is suggested
its on the wiki
its a improvement on my design
[16:03]
JAAAh, neat. [16:04]
IglooYeah JAA, That's fine
This is a method to update all the "stuff" that's out there
And get off Debian 6.
[16:05]
jrwrya
its alpine + docker
so it sticks to the docker version
I updated the wiki to suggest using the beta version
on the very top
[16:05]
IglooThat's fine, But the ones that have been running *forever* need to be updated.
I also want to automate the "your script is out of date, reboot" thing
And make it just do it if it's out of date
[16:11]
Kazwarriors automatically update
unless you mean manual pipelines?
[16:13]
IglooThey do to a point Kaz, One of mine wouldn't run newer jobs due to $issues
But mainly yes, Fix some of the manual pipelines
[16:16]
Kazah fair enough [16:18]
i did consider actually going through and nuking the warrior and starting fresh, but never got round to it
maybe one day
[16:23]
***blitzed has joined #newsgrabber [16:23]
..... (idle for 20mn)
Kimmer1 has quit IRC (Read error: Connection reset by peer) [16:43]
.......... (idle for 45mn)
Fletcher_ has quit IRC (Read error: Operation timed out)
JAA has quit IRC (Read error: Operation timed out)
JAA has joined #newsgrabber
Caz has joined #newsgrabber
svchfoo1 sets mode: +o JAA
rolfoid has quit IRC (Read error: Operation timed out)
[17:28]
rolfoid has joined #newsgrabber
Fletcher_ has joined #newsgrabber
kepler486 has quit IRC (Read error: Operation timed out)
[17:33]
marvinw has quit IRC (Leaving)
marvinw has joined #newsgrabber
[17:42]
........ (idle for 37mn)
jrwrMaybe we should start having a alert on the older ones to update to the new warrior3 [18:20]
.......... (idle for 46mn)
***Caz is now known as Kepler486 [19:06]
Kimmer has joined #newsgrabber [19:11]
........... (idle for 51mn)
Smiley has quit IRC (Ping timeout: 260 seconds)
Smiley has joined #newsgrabber
[20:02]
jrwr3.5 million records over 15 minutes
my god
no wonder my server couldnt keep up
[20:18]
................ (idle for 1h16mn)
IglooIt's been doing it all day jrwr
Just slightly up from the 130,000 - 150,000 from before
21:32 < NewsDeDup> [DEDUPE STATUS] Records Added: 4,340,473
4.3 million now
4777 added per second
(4,300,000 / 15 / 60)
[21:34]
..................... (idle for 1h42mn)
Kazmy SYS box expired nearly a month ago.. I still have access [23:17]
HCross2Anyone know how to accept 40 or so git pulls at once? [23:21]
JAAGitHub, you mean?
I guess you can do it through the API.
https://developer.github.com/v3/pulls/#merge-a-pull-request-merge-button
[23:22]
HCross2someone has made a ton of services, but each in their own PR [23:23]
Kaz40 tabs
ctrl-tab
HCross2: give me a sec, i'll do it
https://s.kurt.gg/214O2BAy.png
[23:24]
HCross2I tried that, but I had to reload the second one after doing the first one
FYI.. harrycross.me is moving mail host atm
so things email wise may not work
[23:25]
Beginning to see CDN load from the States pick up.. this is goo news
good
[23:38]
Kazi've put rbx1 back to work. will be interesting to see how long it lasts [23:42]
HCross2Dedupe CDN is doing wonders for us
with 2 upload/pack queues going on, we're staying ahead of the storage as well
[23:46]
Kazbringing the dl380 online
just need to find a vm I can migrate to it to thrash things out on
[23:48]

↑back Search ←Prev date Next date→ Show only urls(Click on time to select a line by its url)