#newsgrabber 2018-02-20,Tue

Logs of this channel are not protected. You can protect them by a password.

↑back Search ←Prev date Next date→ Show only urls(Click on time to select a line by its url)


WhoWhatWhen
***qw3rty117 has joined #newsgrabber
qw3rty116 has quit IRC (Read error: Operation timed out)
[04:42]
........................................................ (idle for 4h38mn)
Aoedehttps://pastebin.com/jQdRu7pc
any idea why this happens? requests is installed
[09:24]
Kazinstall requests harder
you might have the python3 version, it'll need python2
python packaging is weird
[09:34]
...................................... (idle for 3h9mn)
Aoedethat did it. but I ran into other issues
Aoede tests some more
[12:43]
DeduplicateWarcExtProcArgs in pipeline.py calls python instead of python2
which seems to break it if system uses python3 by default
arkiver: https://github.com/ArchiveTeam/NewsGrabber-Warrior/blob/master/pipeline.py#L176
can you change that into python2?
[12:51]
JAAarkiver hasn't been around in a while. Kaz? ^
Is it correct that newsgrabber needs both Python 2 and Python 3?
[13:00]
Kazuh [13:00]
JAAPython 2 for pipeline & Co., Python 3 for wpull [13:00]
KazI'm at work atm, will have a look int that link once home
Not sure if the version of wpull we're running needs py2 or py3, I can't remember
[13:00]
JAAI think wpull never supported Python 2 in the first place.
Doesn't newsgrabber use that weird binary from launchpad?
I can send a PR if you want.
[13:01]
Kazyeah, something like that
send a PR if you want - bump version number etc please
[13:03]
Aoedeit shouldn't break the warrior or anything?
here's the current error message btw: https://pastebin.com/63LMvf2V
happens everytime
[13:05]
JAAI doubt it. The python2 executable (symlink) has been around for many years. [13:06]
AoedeI see, thanks [13:06]
Kazif it breaks warrior we'll just switch back to python (rather than python2) and warriors will update again [13:06]
JAAAoede: Out of curiosity, which distribution is that?
Kaz: https://github.com/ArchiveTeam/NewsGrabber-Warrior/pull/11
[13:09]
AoedeJAA: scaleway gentoo image [13:09]
Kazmerged, thanks JAA
Aoede: can you pull + test please?
[13:13]
Aoedeokay [13:15]
Kaz: JAA works now, thanks :-) [13:25]
JAASweet
I see warriors on the tracker with the new version as well, so maybe it didn't break there either. :-)
[13:36]
......... (idle for 43mn)
Kazalways good when we don't break things with a one-line pull request [14:20]
JAAIndeed [14:29]
Igloo14:20 <@Kaz> always good when we don't break things with a one-line pull request
First time for everything
[14:35]
Smileyhmmm, archiveteam-warrior-2 still freezes up for me :/
I can't remember if I actually figured out why it was doing that.
and warrior v3 is.... well.. not doing anything
NFO FINISHED.
INFO Duration: 0:00:48. Speed: 92.0 KiB/s.
INFO Downloaded: 140 files, 8.4 MiB.
INFO Exiting with status 4.
Task was destroyed but it is pending!
task: <Task pending coro=<clean() running at /home/box/wpull/freezer/pyinstaller/wpull_env/lib/python3.4/site-packages/wpull/connection.py:362> wait_for=<Task finished coro=<clean() done, defined at /home/box/wpull/freezer/pyinstaller/wpull_env/lib/python3.4/site-packages/wpull/connection.py:119> result=None> cb=[Task._wakeup()]>
errrr
why is warrior 2 (and so python2) using python3.4 libs?
Python version: 2.6.6 (r266:84292, Dec 27 2010, 00:02:40) [GCC 4.4.5]
Also the old extra characters appearing in the log is still going on:
INFO Fetched \u2018https://static.xx.fbcdn.net/rsrc.php/v3i9cV4/yQ/l/en_GB/3d-bkgrd-btn-27.jpg\u2019: 400 Bad Request. Length: 0 [text/html; charset=UTF-8].
infact I think dedupe on v2 warrior is still broken
[14:38]
...... (idle for 27mn)
JAAwpull uses Python 3.
The pipeline and dedupe scripts use Python 2.
Those "extra characters" are just improperly treated UTF-8 output from wpull.
(I think)
But I like hearing that it's "still" broken. Means I didn't break it. :-P
[15:17]
Kazwarrior2 has been working for me
I've got an instance of it running somewhere
[15:25]
................. (idle for 1h23mn)
Smileyyeah it's my v3 warrior that doesn't 'work'
basically if you ever click newsgrabber.... it never quits the project, so never starts somethjing else.
[16:48]
.......................... (idle for 2h6mn)
KazYeah, at this point it's not worth the effort of supporting warrior v3 [18:54]
.......... (idle for 46mn)
JensI'm moderatly certain that wpull uses some bizarre compiled-in python version.
I'm also moderately certain that this is the cause of a lot of random problems.
Why is Newsgrabber Python 2 anyway?
[19:40]
........ (idle for 39mn)
JAABecause it uses a custom version of warcio which doesn't support Python 3, I believe. [20:20]

↑back Search ←Prev date Next date→ Show only urls(Click on time to select a line by its url)