#newsgrabberbot 2017-11-09,Thu

Logs of this channel are not protected. You can protect them by a password.

↑back Search ←Prev date Next date→ Show only urls(Click on time to select a line by its url)


WhoWhatWhen
***HCLUD1 has joined #newsgrabberbot [07:08]
HCLUD1Hello!
Version 20161124.01.
[07:08]
***IGLDN1 has joined #newsgrabberbot [07:08]
IGLDN1Hello!
Version 20161124.01.
[07:08]
***Igloo has quit IRC (Read error: Operation timed out)
CHANFIX has joined #newsgrabberbot
services.int sets mode: +o CHANFIX
CHANFIX sets mode: +o arkiver
[07:09]
CHANFIX1 client should have been opped. [07:09]
***CHANFIX has left
Fletcher_ has quit IRC (Read error: Operation timed out)
[07:09]
........ (idle for 38mn)
CHANFIX has joined #newsgrabberbot
services.int sets mode: +o CHANFIX
CHANFIX sets mode: +oo jrwr Kaz
[07:49]
CHANFIX2 clients should have been opped. [07:49]
***CHANFIX has left [07:49]
..... (idle for 20mn)
CHANFIX has joined #newsgrabberbot
services.int sets mode: +o CHANFIX
CHANFIX sets mode: +o JensRex
[08:09]
CHANFIX1 client should have been opped. [08:09]
***CHANFIX has left [08:09]
................. (idle for 1h20mn)
HCSIND1 has quit IRC (Ping timeout: 248 seconds) [09:29]
Fletcher_ has joined #newsgrabberbot
Igloo has joined #newsgrabberbot
[09:35]
........................................................................................................ (idle for 8h38mn)
jrwr3223 URLs added in the last 15 minutes. [18:15]
NewsDeDup[DEDUPE STATUS] 4 threads alive! - Req Time: 0.09 Total Req: 640,495,657 Req/Sec: 224.81 Records Added: 106,184,239 DB Size: 19.66GB Load: 8% [18:15]
........ (idle for 39mn)
jrwr!status [18:54]
HCLUD1jrwr: Discoverer is running. [18:54]
IGLDN1jrwr: Discoverer is running. [18:54]
IGNYCD01jrwr: Discoverer is running. [18:54]
jrwr!help [18:55]
....... (idle for 33mn)
***newsbuddy has joined #newsgrabberbot [19:28]
newsbuddyHello!
Version 20170617.01.
[19:28]
jrwr!status [19:28]
IGLDN1jrwr: Discoverer is running. [19:28]
HCLUD1jrwr: Discoverer is running. [19:28]
IGNYCD01jrwr: Discoverer is running. [19:28]
newsbuddyFound 936 new services [19:28]
IGLDN1125 services assigned. [19:29]
IGNYCD01124 services assigned. [19:29]
HCLUD1125 services assigned. [19:29]
jrwr3223 URLs added in the last 15 minutes. [19:31]
NewsDeDup[DEDUPE STATUS] 4 threads alive! - Req Time: 0.09 Total Req: 641,415,332 Req/Sec: 206.32 Records Added: 0 DB Size: 19.66GB Load: 10% [19:31]
jrwr3223 URLs added in the last 15 minutes. [19:31]
NewsDeDup[DEDUPE STATUS] 4 threads alive! - Req Time: 0.09 Total Req: 641,417,650 Req/Sec: 207.81 Records Added: 0 DB Size: 19.66GB Load: 12% [19:31]
jrwrholy shit HCross2
dat req/s
[19:32]
IglooAny stats on the high hitters? [19:32]
jrwryou mean most requested?
or most requestees
I wouldn't be able to keep those kinds of logs
[19:33]
HCross2jrwr: are you still looking for more SSD space etc for this? [19:33]
jrwrno
its on spinning disks
and still fast as fuck
I love SSDB
[19:33]
HCross2:) [19:35]
jrwras a K/V store it has blown my mind
its faster then redis at gets, (about 0.25 slower in sets)
https://github.com/ideawu/ssdb
[19:35]
HCross2jrwr: ive had an idea for an experiment id like to try
passing dedupe through a caching CDN
[19:37]
jrwrOh? [19:38]
IglooThat only adds complexity / delay? [19:39]
jrwrI can setup a pub/sub deal were SSDB is ran local to the worker
since it doesn't use much ram
[19:39]
HCross2eh, it was just a brain spew I was having [19:39]
IglooIt could work, But we'd need metrics for testing? [19:40]
HCross2I've got some BunnyCDN credits we could use to test [19:41]
IglooSince OVH's outage earlier, My speed has gone up 1000% between my JOE dedicated and my kimsufi
Now pinning 100Mbps vs before it was 1Mbps
[19:43]
jrwrlol
so if I did a single file as the key and the value as the contents of the file
and do all the tings
things
[19:44]
HCross2give http://NewsGrabberDedupe.b-cdn.net/
a whirl
[19:47]
newsbuddy1361564 URLs added in the last 15 minutes. [19:47]
NewsDeDup[DEDUPE STATUS] 4 threads alive! - Req Time: 0.09 Total Req: 641,614,537 Req/Sec: 207.75 Records Added: 0 DB Size: 19.66GB Load: 7% [19:47]
jrwrtotal_calls
6216078950
thats in the DB
lol
anyway you are welcome to replace the URL for dedupe in your workers and test
[19:50]
HCross2im just curious really [19:51]
jrwrya
you have all the workers running
go drop one and see :)
[19:51]
HCross2all my workers are 1 dedi going flat out [19:52]
jrwrlol [19:52]
IglooI'm keeping an eye on my bandwidth, I have 20TB and need to make sure I don't go over it ;) [19:53]
jrwrwould be cool to drop it in as a default for the warriors [19:53]
HCross2I may do it.. then watch my BunnyCDN account and see what it costs
jrwr: hit ratio isnt actually that abd
bad
[19:55]
jrwr3223 URLs added in the last 15 minutes. [20:01]
NewsDeDup[DEDUPE STATUS] 4 threads alive! - Req Time: 0.09 Total Req: 641,727,357 Req/Sec: 119.49 Records Added: 0 DB Size: 19.66GB Load: 10% [20:01]
newsbuddy0 URLs added in the last 15 minutes. [20:02]
NewsDeDup[DEDUPE STATUS] 4 threads alive! - Req Time: 0.09 Total Req: 641,735,209 Req/Sec: 112.17 Records Added: 0 DB Size: 19.66GB Load: 7% [20:02]
.... (idle for 16mn)
newsbuddy0 URLs added in the last 15 minutes. [20:18]
NewsDeDup[DEDUPE STATUS] 4 threads alive! - Req Time: 0.09 Total Req: 641,849,068 Req/Sec: 131.79 Records Added: 0 DB Size: 19.66GB Load: 11% [20:18]
.... (idle for 15mn)
newsbuddy0 URLs added in the last 15 minutes.
Warriorlists failed to upload to tracker.
[20:33]
NewsDeDup[DEDUPE STATUS] 4 threads alive! - Req Time: 0.09 Total Req: 641,963,378 Req/Sec: 114.02 Records Added: 0 DB Size: 19.66GB Load: 16% [20:33]
.... (idle for 15mn)
newsbuddy0 URLs added in the last 15 minutes. [20:48]
NewsDeDup[DEDUPE STATUS] 4 threads alive! - Req Time: 0.09 Total Req: 642,082,130 Req/Sec: 117.49 Records Added: 0 DB Size: 19.66GB Load: 6% [20:48]
.... (idle for 15mn)
newsbuddy0 URLs added in the last 15 minutes.
Warriorlists failed to upload to tracker.
[21:03]
NewsDeDup[DEDUPE STATUS] 4 threads alive! - Req Time: 0.09 Total Req: 642,199,598 Req/Sec: 129.07 Records Added: 0 DB Size: 19.66GB Load: 8% [21:03]
.... (idle for 15mn)
newsbuddy0 URLs added in the last 15 minutes. [21:18]
NewsDeDup[DEDUPE STATUS] 4 threads alive! - Req Time: 0.09 Total Req: 642,304,837 Req/Sec: 107.69 Records Added: 0 DB Size: 19.66GB Load: 6% [21:18]
.... (idle for 15mn)
newsbuddy0 URLs added in the last 15 minutes.
Warriorlists failed to upload to tracker.
[21:33]
NewsDeDup[DEDUPE STATUS] 4 threads alive! - Req Time: 0.09 Total Req: 642,403,380 Req/Sec: 112.29 Records Added: 0 DB Size: 19.66GB Load: 5% [21:33]
.... (idle for 15mn)
newsbuddy0 URLs added in the last 15 minutes. [21:48]
NewsDeDup[DEDUPE STATUS] 4 threads alive! - Req Time: 0.09 Total Req: 642,517,128 Req/Sec: 122.17 Records Added: 0 DB Size: 19.66GB Load: 9% [21:48]
.... (idle for 15mn)
newsbuddy0 URLs added in the last 15 minutes.
Warriorlists failed to upload to tracker.
[22:03]
NewsDeDup[DEDUPE STATUS] 4 threads alive! - Req Time: 0.09 Total Req: 642,624,576 Req/Sec: 114.88 Records Added: 0 DB Size: 19.66GB Load: 16% [22:03]
.... (idle for 15mn)
newsbuddy0 URLs added in the last 15 minutes. [22:18]
NewsDeDup[DEDUPE STATUS] 4 threads alive! - Req Time: 0.09 Total Req: 642,731,577 Req/Sec: 117.99 Records Added: 0 DB Size: 19.66GB Load: 17% [22:18]
.... (idle for 15mn)
newsbuddy0 URLs added in the last 15 minutes.
Warriorlists failed to upload to tracker.
[22:33]
NewsDeDup[DEDUPE STATUS] 4 threads alive! - Req Time: 0.09 Total Req: 642,843,389 Req/Sec: 122.67 Records Added: 0 DB Size: 19.66GB Load: 27% [22:33]
.... (idle for 15mn)
newsbuddy0 URLs added in the last 15 minutes. [22:48]
NewsDeDup[DEDUPE STATUS] 4 threads alive! - Req Time: 0.09 Total Req: 642,954,604 Req/Sec: 111.21 Records Added: 217,321 DB Size: 19.68GB Load: 62% [22:48]
.... (idle for 15mn)
newsbuddy0 URLs added in the last 15 minutes.
Warriorlists failed to upload to tracker.
[23:03]
NewsDeDup[DEDUPE STATUS] 4 threads alive! - Req Time: 0.09 Total Req: 643,057,916 Req/Sec: 121.98 Records Added: 357,234 DB Size: 19.68GB Load: 69% [23:03]
.... (idle for 15mn)
newsbuddy0 URLs added in the last 15 minutes. [23:18]
NewsDeDup[DEDUPE STATUS] 4 threads alive! - Req Time: 0.09 Total Req: 643,157,154 Req/Sec: 104.86 Records Added: 358,522 DB Size: 19.69GB Load: 56% [23:18]
.... (idle for 15mn)
newsbuddy0 URLs added in the last 15 minutes.
Warriorlists failed to upload to tracker.
[23:33]
NewsDeDup[DEDUPE STATUS] 4 threads alive! - Req Time: 0.09 Total Req: 643,260,410 Req/Sec: 123.84 Records Added: 279,980 DB Size: 19.69GB Load: 67% [23:33]
.... (idle for 15mn)
newsbuddy0 URLs added in the last 15 minutes. [23:48]
NewsDeDup[DEDUPE STATUS] 4 threads alive! - Req Time: 0.09 Total Req: 643,365,868 Req/Sec: 120.65 Records Added: 569,673 DB Size: 19.71GB Load: 57% [23:48]

↑back Search ←Prev date Next date→ Show only urls(Click on time to select a line by its url)