arkiver: we are full and not uploading for some reason Hello! I've just been (re)started. Follow my newsgrabs in #newsgrabberbot Hello! I've just been (re)started. Follow my newsgrabs in #newsgrabberbot HCross2: HCross: most of the space on the server is currently taking by incomplete WARCs from the .rsync-tmp directories of older projects ahh ok - recon we could just delete them, or do some sort of "fix and upload" For example for panoramio there's 1.1 TB of incomplete WARCs we could delete them but there's probably some records in the WARCs that have been fully synced 1.1 TB is quite a lot of good records probably yeah, we need to do something to preseve them yep Can we go through record by record and check it for validity or is there a better way? I hope we can do that Hello! I've just been (re)started. Follow my newsgrabs in #newsgrabberbot arkiver, if I was to arrange somewhere to shove the temp files off to, how much space would we need? I'm not sure a few TB would be good to clear quite some space but it might not be needed for some reason moving of WARCs from newsbuddy is going very slow let's fix that first ill have a look at the network side now ok arkiver: see slack MTRs outbound look fine we dont have the switch issues weve had last time there's a slack? I've got 8TB free on 1Gbps, 20T free on 0.5Gbps. let me know if you want to store something ~ALERT~ Dedupe is not getting any requests yes, this is not running at the moment https://nypost.com/2017/08/03/salon-struggling-to-pay-its-rent/ www.salon.com\/20[0-9]{2}/[0-9]{2}/[0-9]{2}/.+ for url www.salon.com\/20[0-9]{2}\/[0-9]{2}\/[0-9]{2}\/.+