#newsgrabber 2017-09-16,Sat

Logs of this channel are not protected. You can protect them by a password.

↑back Search ←Prev date Next date→ Show only urls(Click on time to select a line by its url)


WhoWhatWhen
***VADemon_ has joined #newsgrabber
VADemon has quit IRC (Ping timeout: 255 seconds)
[04:27]
...................................................................... (idle for 5h47mn)
HCross2arkiver: we are full and not uploading for some reason [10:16]
................................................................. (idle for 5h20mn)
***newsbuddy has quit IRC (Read error: Connection reset by peer)
newsbuddy has joined #newsgrabber
[15:36]
newsbuddyHello! I've just been (re)started. Follow my newsgrabs in #newsgrabberbot [15:36]
***newsbuddy has quit IRC (Remote host closed the connection)
newsbuddy has joined #newsgrabber
[15:38]
newsbuddyHello! I've just been (re)started. Follow my newsgrabs in #newsgrabberbot [15:41]
arkiverHCross2: HCross: most of the space on the server is currently taking by incomplete WARCs from the .rsync-tmp directories of older projects [15:45]
HCross2ahh ok - recon we could just delete them, or do some sort of "fix and upload" [15:45]
arkiverFor example for panoramio there's 1.1 TB of incomplete WARCs
we could delete them
but there's probably some records in the WARCs that have been fully synced
1.1 TB is quite a lot of good records probably
[15:45]
HCrossyeah, we need to do something to preseve them [15:48]
arkiveryep [15:48]
HCrossCan we go through record by record and check it for validity or is there a better way? [15:49]
arkiverI hope we can do that [15:50]
***newsbuddy has quit IRC (Remote host closed the connection)
newsbuddy has joined #newsgrabber
[15:50]
newsbuddyHello! I've just been (re)started. Follow my newsgrabs in #newsgrabberbot [15:50]
***newsbuddy has quit IRC (Remote host closed the connection) [15:52]
HCrossarkiver, if I was to arrange somewhere to shove the temp files off to, how much space would we need? [15:54]
.... (idle for 16mn)
arkiverI'm not sure
a few TB would be good to clear quite some space
but it might not be needed
for some reason moving of WARCs from newsbuddy is going very slow
let's fix that first
[16:10]
HCross2ill have a look at the network side now [16:26]
arkiverok [16:27]
.... (idle for 18mn)
HCross2arkiver: see slack
MTRs outbound look fine
we dont have the switch issues weve had last time
[16:45]
........ (idle for 39mn)
trvzthere's a slack? [17:25]
I've got 8TB free on 1Gbps, 20T free on 0.5Gbps. let me know if you want to store something [17:32]
.................................................................... (idle for 5h39mn)
jrwr~ALERT~ Dedupe is not getting any requests [23:11]
arkiveryes, this is not running at the moment [23:18]
...... (idle for 26mn)
***dd0a13f37 has joined #newsgrabber [23:44]
icedice has joined #newsgrabber [23:52]
icedicehttps://nypost.com/2017/08/03/salon-struggling-to-pay-its-rent/ [23:53]
dd0a13f37www.salon.com\/20[0-9]{2}/[0-9]{2}/[0-9]{2}/.+ for url
www.salon.com\/20[0-9]{2}\/[0-9]{2}\/[0-9]{2}\/.+
[23:59]

↑back Search ←Prev date Next date→ Show only urls(Click on time to select a line by its url)