#newsgrabber 2018-04-21,Sat

Logs of this channel are not protected. You can protect them by a password.

↑back Search ←Prev date Next date→ Show only urls(Click on time to select a line by its url)


WhoWhatWhen
***MrRadar has quit IRC (Read error: Operation timed out)
dxrt has quit IRC (Read error: Operation timed out)
Igloo_ has joined #newsgrabber
Igloo has quit IRC (Write error: Broken pipe)
Atom has quit IRC (Read error: Operation timed out)
dxrt has joined #newsgrabber
MrRadar has joined #newsgrabber
[03:23]
..... (idle for 23mn)
qw3rty119 has joined #newsgrabber [03:54]
qw3rty118 has quit IRC (Read error: Operation timed out) [04:00]
......................................................... (idle for 4h43mn)
HCrossarkiver: follow up from the panic yesterday, we've been stable overnight [08:43]
..................................................................................................................................................... (idle for 12h22mn)
***Atom has joined #newsgrabber [21:05]
Atom-- has joined #newsgrabber
Atom has quit IRC (Read error: Operation timed out)
[21:10]
..... (idle for 22mn)
odemgHCross, how many sites does this project grab from? Are we only grabbing new content or are we archiving historical content too, I'm asking because while this is an ongoing project with potentially unlimited data to grab we're moving generally rather slow, but is there reason to speed up, can we speed up, what are our first bottlenecks if we do speed up? [21:36]
HCrossWe're getting approximately 1000 source sites, grabbing new content and old due to our backlog. We can speed up, but we need more concurrent (warrior was good at this), but our first bottleneck I see is potentially my upload server [21:45]
Kazhttps://github.com/ArchiveTeam/NewsGrabber/tree/master/services odemg [21:55]

↑back Search ←Prev date Next date→ Show only urls(Click on time to select a line by its url)