#newsgrabber 2018-03-07,Wed

Logs of this channel are not protected. You can protect them by a password.

↑back Search ←Prev date Next date→ Show only urls(Click on time to select a line by its url)


WhoWhatWhen
***qw3rty115 has joined #newsgrabber [04:10]
qw3rty114 has quit IRC (Read error: Operation timed out) [04:15]
.............................................................................................................................................. (idle for 11h45mn)
Meroje has quit IRC (Quit: bye!) [16:00]
Meroje has joined #newsgrabber [16:10]
....... (idle for 33mn)
Atom--when using the NewsGrabber-Warrior scripts, how do I specify the number of dedupe and rsync thread... is there a command line option, or just modify the pipeline file? [16:43]
Aoedeafaik there isn't a way to do it. don't edit the script
you could start multiple instances of the script though
[16:45]
Atom--in the warrior there were config options for both, but since the warrior is broken I'm just cloning the existing scripts and running them. The default values are listed in the pipeline.py file... just wondering if I can set it via command line or env [16:48]
***odemg has quit IRC (Read error: Operation timed out) [17:00]
.... (idle for 17mn)
odemg has joined #newsgrabber [17:17]
............. (idle for 1h1mn)
SmileyGAtom--: the warrior runs multiple instances of the script.
if you want i can give you a bash script to launch X instances
[18:18]
in pipelinedir:
for x in 1..100; do screen -S $x ./pipeline...; done
[18:24]
.......................................... (idle for 3h27mn)
KazAtom--: There is a way to do it via a command line option, but I don't remember it atm. You *can* change the 'default' value on lines 309 + 319 to get the amount of threads you want
As long as you don't edit anything else in the script
[21:51]
.... (idle for 16mn)
***vantec has quit IRC (Quit: Time wasted: $rand(30) days $rand(23) hours $rand(59) minutes $rand(59) seconds $rand(99) milliseconds ) [22:08]
.............. (idle for 1h5mn)
Atom--Kaz, ty [23:13]

↑back Search ←Prev date Next date→ Show only urls(Click on time to select a line by its url)