[12:14] <ivan`> underscor: during the upcoming greader grab, would it be feasible to run queries over all the warc.gz's uploaded to the server to find more URLs?
[12:20] <Smiley> ivan`: we've had warriors report new usernames before.
[12:20] <Smiley> No reaason why we can't report new urls.
[12:20] <Smiley> *that I know of*
[12:20] <Smiley> ersi: check it's not 0bytes etc.
[12:20] <Smiley> just some really basic validation.
[12:20] <ivan`> I was thinking about putting this into the greader-grab pipeline, but it's going to delay things by a day or two
[12:20] <ivan`> and I don't really know which URLs I want yet
[12:20] <ivan`> I know some, but not all
[12:23] <ersi> Smiley: Uh, yeah - if you don't remember: You wrote the exact same line yesterday evening.
[12:29] <Smiley> yeah I rmemeber
[12:29] <Smiley> :D