#newsgrabber 2017-08-26,Sat

Logs of this channel are not protected. You can protect them by a password.

↑back Search ←Prev date Next date→ Show only urls(Click on time to select a line by its url)


WhoWhatWhen
***SK1d_ has joined #newsgrabber [00:23]
SK1d_I try to help the grab by running the warrior but somehow this does not work is this a known problem?
I have the task selected but no new windows appear unter current project. If i switch to the yahoo project it works.
[00:26]
.................................................. (idle for 4h7mn)
***SK1d_ has quit IRC (Ping timeout: 268 seconds) [04:34]
........................................................................................... (idle for 7h31mn)
YortMeist has joined #newsgrabber [12:05]
...... (idle for 25mn)
YortMeist has quit IRC () [12:30]
YortMeist has joined #newsgrabber [12:38]
.............................. (idle for 2h26mn)
HCross2arkiver: im going to start seeing if I can write a cloud-init script for newsbuddy setup [15:04]
arkiverwhat is that?
I'm having a look at the strange warrior problems this project has
[15:12]
HCross2arkiver: https://cloud-init.io so you can specify a script on a cloud provider as you spin up an instance, and the first thing the instance will do is that script. So Ill be able to bring up more warriors by just telling them to start, and theyll boot themselves up, download and install all the warrior code and then start grabbing - without intervention [15:15]
arkiverthat's interesting [15:15]
HCross2mainly its just specifying a bash file - which is then run [15:16]
arkiverwould make it much easier for other to set everything up too [15:17]
HCross2yep - if their provider/whoever supports cloud-init - they just shove in the URL of the script and it does things [15:18]
arkiverI see many distributions are supported [15:19]
HCross2we could really do with more grabbers [15:22]
arkiverif warrior are working good and the wiki is finished we can place some messages on datahoarders
huh the warrior uses python2.6
well that explain why it didn't work
[15:25]
.... (idle for 19mn)
HCross2Just thinking.. for hosts that dont do cloud-init.. I can always just fire off an SSH command from another server that logs into the destination as root and then fires off my script
hm. It takes a warrior instance about 4 minutes to get down to a reasonable load once you first kick off a high concurrent
[15:50]
arkiverwarrior works now!
minimum in tracker is now 20170826.01
[15:56]
HCross2ok
arkiver: you SSH'd into master at all? My session just locked up
I can reconnect though
[15:57]
arkiverI can login [15:59]
HCross2ive got a feeling it was my end [16:00]
@newsbuddy.net email accounts are available if people want one.. just let me know [16:11]
...... (idle for 28mn)
so far.. I have a script that wont stop spawning instances [16:39]
Anyone able to lend me a hand with this script please? [16:51]
................ (idle for 1h17mn)
trvznice, your graphs are growing again [18:08]
...... (idle for 29mn)
HCross2jrwr: you enjoying my dedupe requests :p [18:37]
.......... (idle for 45mn)
FYI ALL: Upload speeds may be slow again... I think OVH have a switch issue on the rack where newsbuddy is
Ill ring OVH on Monday
[19:22]
***mls has quit IRC (Ping timeout: 250 seconds) [19:27]
jrwrso
HCross2:
You only made a small dent in the load
from 4% CPU to 5% Cpu
[19:30]
HCross2haha - can you tell its all me [19:30]
jrwrLoop delay 0.09ms
0.09MS
BET THAT BITCHES
Dude SSDB is insane for a database
[19:31]
HCross2jrwr: if its AS9009 traffic.. its probably me [19:31]
jrwrI don't keep those kind of logs
mostly just this
https://usercontent.irccloud-cdn.com/file/qpyz8Q60/image.png
[19:31]
HCross2ahh [19:32]
jrwrthat current req processed is per thread
so times that by 4
Its a disk based K:V Store
yet faster then in ram Redis
WTF
Ah
[19:33]
HCross2is it just me.. or is this more ironic than it should be https://usercontent.irccloud-cdn.com/file/awHDhnhs/image.png [19:35]
jrwrI just want a 100Gbit pipe to my house [19:36]
.......................... (idle for 2h7mn)
JAAHaha, nice find. [21:43]

↑back Search ←Prev date Next date→ Show only urls(Click on time to select a line by its url)