#newsgrabber 2017-11-19,Sun

Logs of this channel are not protected. You can protect them by a password.

↑back Search ←Prev date Next date→ Show only urls(Click on time to select a line by its url)


WhoWhatWhen
***Martle_ has joined #newsgrabber
Martle has quit IRC (Read error: Operation timed out)
[00:20]
Martle__ has joined #newsgrabber [00:29]
Martle_ has quit IRC (Read error: Operation timed out) [00:36]
.................... (idle for 1h36mn)
kyan has joined #newsgrabber [02:12]
............ (idle for 58mn)
dd0a13f37 has quit IRC (Quit: Connection closed for inactivity) [03:10]
............................................... (idle for 3h51mn)
kyan has quit IRC (Quit: Leaving) [07:01]
..................... (idle for 1h43mn)
anonymoos has joined #newsgrabber [08:44]
anonymooscdn dedup seems to be getting stuck [08:46]
.................................................................................................................. (idle for 9h27mn)
***Martle has joined #newsgrabber
Martle__ has quit IRC (Read error: Operation timed out)
[18:13]
......................... (idle for 2h0mn)
blitzed has joined #newsgrabber [20:15]
.... (idle for 16mn)
JensRexA question to people who are running an absurd number of pipelines. How are you babysitting everything?
I have to kill off the pipeline like once every day or two because it's stuck.
[20:31]
KazJensRex: i'm not running a huge amount, but the logic still applies
*/30 * * * * touch /home/kurt/NewsGrabber-Warrior/STOP && cd /home/kurt/NewsGrabber-Warrior && /home/kurt/NewsGrabber-Warrior/screen.sh >/dev/null 2>&1
screen.sh just kicks off 15 threads at concurrency 1
[20:33]
JensRexI think it was JAA that had made a script some time ago to see when the pipeline had wedged itself. I think it was during appnet days.
It's a bit annoying though. URLteam I can just leave running forever and not worry about it.
Yesterday I was messing around with a VM running 4 threads in four virtual terminals at concurrency 4.
When I kicked one pipeline in the teeth, somehow another one unstuck itself. It was very odd.
[20:35]
HCross2probably hit a disk IO cap or something
or a request cap on a distant site
[20:37]
Kazi've been a ton of them lock up, i just pop in every few hours and kill off old sessions
i haven't worked out what the cause is, seen things across various sites
[20:38]
JensRexI'm still wondering about this bit from pipeline.py: '--session-timeout', str(86400 * 2),
That's 48 hour timeout for... something.
[20:38]
Kazhah [20:39]
***blitzed has quit IRC (Quit: Leaving) [20:39]
Kazi've just changed it locally to 1800 * 2
we'll see how it goes
[20:39]
JensRexDon't let a-rkiver know... he'll be furious :D [20:40]
Kazpfft
it's all about modifying seesaw and making yipdw mad
my pipelines shouldn't really last much more than 45 minutes *max* anyway
[20:40]
HCross2stop changing the script [20:42]
JensRexOh shit. Five-O five-o! [20:43]
HCross2JensRex: send me some good quality danish bacon and I wont tell anyone :p [20:44]
JensRexI've changed nothing. I already got chewed out once by arkiver for doing that :) [20:44]
Kazhttps://s.kurt.gg/1a2J87b8.png [20:44]
JensRexI just gently shoot the pipeline in the face once a day. [20:45]
Kazthat's the extent of my changes [20:45]
JensRexI'd just like to know why there's a 48 hour timeout. Maybe it makes sense for some reason I haven't thought of. [20:45]
Kazit might do.. but if I've got any pipelines running for 48h we've got a bigger issue [20:46]
JensRexYes. [20:46]
JAAYep, I wrote something to kill stuck wget-lua processes for appnet, I believe. I'm not sure if I still have the code though. [20:48]
JensRexI tried increasing verbosity on wpull to figure out where and why it wedged itself, but I learned nothing.
JAA: I just found it. https://bpaste.net/show/4b53ab30193e
[20:48]
JAAJensRex: Ah yeah, that's an earlier version. I think I changed it to kill the process automatically.
I posted it to https://ghostbin.com/paste/t9wat on 2017-03-16, but that link is unfortunately dead by now.
[20:49]
JensRexArchiveTeam lost a URL... [20:52]
JAAHeh
Found it in my notes.
https://hastebin.com/inudabixag.bash
It kills wget-lua processes which have been stuck for 10 minutes (according to the log file).
[20:53]
...... (idle for 27mn)
***trvz has quit IRC (Ping timeout: 260 seconds) [21:27]
................ (idle for 1h16mn)
matt_ has joined #newsgrabber
matt_ is now known as Igloo_
Igloo_ has quit IRC (Client Quit)
[22:43]
Igloo_ has joined #newsgrabber
Igloo has quit IRC (Quit: leaving)
Igloo_ is now known as Igloo
[22:52]
.......... (idle for 46mn)
trvz has joined #newsgrabber [23:40]

↑back Search ←Prev date Next date→ Show only urls(Click on time to select a line by its url)