I have a sneaking suspicion that the precompiled wpull is kind of poopy.
"ERROR Fetching \u2018https://site.com/example.js\u2019 encountered an error: Connect timed out."
That's never going to work. Invalid URL.
Looks valid to me? \u2018 and \u2019 are the markers around the URL, ‘URL’.
Something's broken about those two characters, obviously.
I know, but it looks like someone used it as an example url.
I don't know if the job failed because of that.
not much anyone can do about that, that URL will be tried a few times then fail, and wpull will move on to the next one
people put all sorts of bizarre shit into their sites, nothing to worry about too much
Okay good. It just got hung up there for a long while. Thought it might fail out.
I'm not really familiar with NewsGrabber, is it using wpull 2.0.x?
can't remember what the timeout is for stuff like that, could be up to like 5 minutes
No, 1.2.3.
Ah ok
considering site.com does have a valid A record
pipeline.py is Python2, so it can't use regular wpull because that's Python3 only. And the precompiled wpull is weird.
https://github.com/ArchiveTeam/NewsGrabber-Warrior/blob/31b29fc0ba61a05b081c06852d9ae9413421cab5/pipeline.py#L254-L259
Ah right, same as the Flickr project.
We really need to move this to Python 3.
It's been almost 10 years since that was released...
Hmm, investigating youtube-dl failure further: https://bpaste.net/show/73eb42a23506
I'm not sure what to make of that.
Newsgrabber fails on every video job here.
Python 2 vs. 3?
That paste is clearly using Python 3...
That's just using the precompiled wpull.
Uninstalled local youtube-dl --> https://bpaste.net/show/a7fe1552962c
Hmm
So that "precompiled" wpull is Python 3.4, as far as I can tell.
I wonder though, why can't you run the normal wpull with Python 3? pipeline.py doesn't "import wpull", it just executes a subprocess.
I tried that yesterday in a test VM. It failed spectacularly.