#newsgrabber 2017-06-20,Tue

Logs of this channel are not protected. You can protect them by a password.

↑back Search ←Prev date Next date→ Show only urls(Click on time to select a line by its url)


WhoWhatWhen
***ivan has quit IRC (Leaving)
ivan has joined #newsgrabber
[00:23]
.... (idle for 19mn)
kyan has quit IRC (Read error: Operation timed out) [00:43]
blitzed has quit IRC (Quit: Leaving)
blitzed has joined #newsgrabber
[00:50]
....................................... (idle for 3h14mn)
MrRadar2 has joined #newsgrabber [04:08]
..................... (idle for 1h42mn)
Aranje has joined #newsgrabber [05:50]
...... (idle for 28mn)
Aranje has quit IRC (Quit: Three sheets to the wind)
Aranje has joined #newsgrabber
[06:18]
.... (idle for 17mn)
Aranje has quit IRC (Three sheets to the wind) [06:36]
..................................................... (idle for 4h23mn)
underscor has joined #newsgrabber [10:59]
underscorI notice the newsgrabber scripts are downloading fairly large videos with youtube-dl, but the completed tasks seem to be only uploading a few megs at most
What's happening to those videos? Are they not getting backed up?
[11:00]
.... (idle for 19mn)
HCross2arkiver: ^ I've got a hunch they are deduplicating [11:19]
........ (idle for 35mn)
KazI think I'm still having issues with youtube-dl
every job with videos in fails for me, haven't checked logs in a while
[11:54]
.... (idle for 15mn)
***MrRadar has quit IRC (Quit: Shut down for planned power outage) [12:09]
midasKaz: on a ovh host? [12:22]
Kazyes [12:23]
midasseems that youtube has blocked access from ovh
my downloads dont work from ovh but work from my transip vps
[12:24]
Kaz...ah
hmm
I think that's a separate issue actually, wpull can't even find youtubedl for me
midas: are you getting an SSL error from wpull when trying youtube?
[12:24]
midasim getting a not allowed [12:39]
KazWARNING ERROR: Unable to download webpage: <urlopen error [SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed (_ssl.c:600)> (caused by URLError(SSLError(1, '[SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed (_ssl.c:600)'),))
hang on, update for ca-certificates
[12:47]
.................................... (idle for 2h55mn)
***Aranje has joined #newsgrabber [15:43]
jrwrAlso the scripts are wonky on onn0x86 hardware
non-x86
[15:48]
............. (idle for 1h0mn)
underscorKaz: I ended up having to make a symbolic link from /usr/local/bin/youtube-dl to /usr/bin/youtube-dl to fix that [16:48]
Kazhave given that a shot, except I just copied it over because I can't be bothered to work out which way the arguments go with ln
we'll see how it goes, cheers
[16:51]
underscorhaha
cool
HCross2: Yeah I realized that after reading the code - wayback index already has those video files for a lot of them apparently
I got a few whoppers though, noticed a 35GB one slide by lol
[16:51]
JAAKaz: That's easy to remember: it's the same order as the other common file management tools (cp, rsync, mv): ln $existingFile $newLink [16:56]
Kazah is it, I'll try to remember for next time
it's one of those things I don't use often enough to remember :(
[17:00]
arkiverhi underscor
yes, those are very likely deduplicated videos
looks like we're missing some discoverers
[17:02]
not that that matters very much at the moment [17:08]
underscorwhat are discoverers? can I run one? [17:13]
arkiverit's a machine that discovers new URLs
sure
I think
HCross2 manages those
[17:13]
underscorAh ok [17:14]
arkiveryou should ask him [17:14]
underscorwill do [17:14]
HCross2Hi [17:14]
underscorHey
(see above)
[17:14]
HCross2Perfect timing - discoverers use a moderate amount of RAM and not too much network.. mine avg around 600Kbps - 1Mbps, however they can use CPU [17:15]
underscorperfect
I have a box with lots of RAM and CPU but only 5mbit up
[17:15]
HCross2That's fine
You'll see CPU spikes when a site has horrible coding your box has to deal with
underscor: is it your home internet?
[17:16]
........ (idle for 35mn)
underscoryeah
HCross2: well, business class time warner
but it's at home
[17:51]
...... (idle for 29mn)
Kazunderscor just put through a 30.5gb item
I just did a 556mb item
jrwr: here?
[18:21]
underscorwoah that's a big item [18:28]
........................... (idle for 2h13mn)
***midas1 has quit IRC (ircd.choopa.net irc.mzima.net)
kurt has quit IRC (ircd.choopa.net irc.mzima.net)
randomdes has quit IRC (ircd.choopa.net irc.mzima.net)
[20:41]
midas1 has joined #newsgrabber
kurt has joined #newsgrabber
randomdes has joined #newsgrabber
irc.mzima.net sets mode: +o kurt
[20:49]
.......... (idle for 49mn)
kyan has joined #newsgrabber [21:38]
............... (idle for 1h13mn)
MrRadar has joined #newsgrabber [22:51]

↑back Search ←Prev date Next date→ Show only urls(Click on time to select a line by its url)