arkiver: we are full and not uploading for some reason
Hello! I've just been (re)started. Follow my newsgrabs in #newsgrabberbot
Hello! I've just been (re)started. Follow my newsgrabs in #newsgrabberbot
HCross2: HCross: most of the space on the server is currently taking by incomplete WARCs from the .rsync-tmp directories of older projects
ahh ok - recon we could just delete them, or do some sort of "fix and upload"
For example for panoramio there's 1.1 TB of incomplete WARCs
we could delete them
but there's probably some records in the WARCs that have been fully synced
1.1 TB is quite a lot of good records probably
yeah, we need to do something to preseve them
yep
Can we go through record by record and check it for validity or is there a better way?
I hope we can do that
Hello! I've just been (re)started. Follow my newsgrabs in #newsgrabberbot
arkiver, if I was to arrange somewhere to shove the temp files off to, how much space would we need?
I'm not sure
a few TB would be good to clear quite some space
but it might not be needed
for some reason moving of WARCs from newsbuddy is going very slow
let's fix that first
ill have a look at the network side now
ok
arkiver: see slack
MTRs outbound look fine
we dont have the switch issues weve had last time
there's a slack?
I've got 8TB free on 1Gbps, 20T free on 0.5Gbps. let me know if you want to store something
~ALERT~ Dedupe is not getting any requests
yes, this is not running at the moment
https://nypost.com/2017/08/03/salon-struggling-to-pay-its-rent/
www.salon.com\/20[0-9]{2}/[0-9]{2}/[0-9]{2}/.+  for url
www.salon.com\/20[0-9]{2}\/[0-9]{2}\/[0-9]{2}\/.+