#newsgrabber 2018-01-02,Tue

Logs of this channel are not protected. You can protect them by a password.

↑back Search ←Prev date Next date→ Show only urls(Click on time to select a line by its url)


WhoWhatWhen
***ivan has quit IRC (Leaving)
marvinw has joined #newsgrabber
[01:45]
....................... (idle for 1h52mn)
blitzed has quit IRC (Remote host closed the connection) [03:38]
................................................. (idle for 4h1mn)
Martle has quit IRC (Read error: Connection reset by peer) [07:39]
..... (idle for 24mn)
Martle has joined #newsgrabber [08:03]
BnARobin_ has joined #newsgrabber
BnARobin has quit IRC (Read error: Operation timed out)
[08:12]
............................... (idle for 2h33mn)
steve_ has joined #newsgrabber
steve_ has quit IRC (Client Quit)
[10:46]
........................... (idle for 2h13mn)
jrwrHCross2: whats the old IP
ill start porting it
[12:59]
.................................................. (idle for 4h6mn)
***blitzed has joined #newsgrabber [17:05]
HCross2jrwr: 194.79.31.56 [17:12]
...... (idle for 27mn)
JAAHaha HCross2, I just went through the "2017 Provider of the Year" thread on LET and saw this comment: "M247, for being ambitious." [17:39]
HCross2The "we tried" award
JAA: it's fun running that shitfest
[17:39]
.... (idle for 19mn)
Kazrighty ho
lets do newsgrabber updates
[17:58]
Igloofire fire fire [17:59]
Kazright
you mentioned HCross2 had an update too?
[17:59]
IglooYeah, Pull #3
I haven't tested it
[18:00]
Kazwho the hell is jcj83429
ah, it's anonymoos
[18:00]
Igloo: can't merge yours after pulling anonymoos' in, as pipeline.py has changed
can you make another pull req with updated version?
[18:06]
IglooOh
Yeah, one mo Kaz
[18:10]
Kazta [18:11]
IglooOh heh i'm a twit anyway
Can you delete that pull request Kaz ?
[18:12]
Kazprobably
update: yes
[18:13]
IglooPull #5 is there now Kaz
Oh let me change the version #
[18:15]
KazI've done it, dw
all merged, going to force updated version in tracker now
[18:16]
IglooRoger doger
Can you update other areas Kaz or just the newsgrabber?
I might go spend some time fixing the warrior upgrade bundle if so
[18:17]
HCross2Kaz: then we can set newsgrabber as default in the tracker :) [18:18]
Kazall of it, all over the shop
HCross2: I don't have global tracker admin
or at least if I do I don't know where to log in
[18:18]
IglooLet me see if I can fix the warrior script then, I'll spend some time tonight as debian 6 is well broke now [18:19]
HCross2Ah nor do I [18:19]
KazIgloo: I know someone tried to push for a docker container, it's available somewhere
I'm assuming it's just exactly the same stuff, but with *buzzwords* and shit
[18:19]
Igloofuckdocker ;-) [18:20]
Smileylets see if i'm still a admin ;D
Kaz: ur global admin
HCross2: you too
[18:20]
Kazthanks :) [18:22]
Smileyenjoy. don't fuck it up [18:22]
Kazhow do I log in? [18:22]
Smileyerrr same way you already did [18:22]
Igloo18:22 < Smiley> enjoy. don't fuck it up | straight into | 18:22 <@Kaz> how do I log in? [18:22]
SmileyIgloo: ;) [18:23]
IglooThings can only get better from here ;-) [18:23]
Smileytho i can't see where i set the default project these days [18:23]
HCross2Going well already then :p [18:23]
Kazoh there's a global admin button [18:23]
Smiley\o/
actually i found it
what project you want as default?
[18:24]
Kazhttps://tracker.archiveteam.org/global-admin/warrior right? [18:24]
Smileyyus [18:24]
Kaznewsgrabber anyway
time to watch master burn to the ground
[18:24]
Smileyboom done [18:25]
Kazta [18:25]
Smiley33Mb/user atm
Jsharp being the only warrior instance in the recent jobs list atm
[18:25]
Kazyeah, I'm waiting for new job requests to show up on the tracker.. [18:26]
Smiley83499out + 583147to do [18:26]
IglooKaz: I made a new pull, i made a mistake on the pip command [18:26]
Smileylulderp [18:26]
Kazyou little [18:26]
Smileyomfg how many trackers [18:27]
IglooHey! Be nice else I won't fix stuff :-) [18:27]
Smileyah, 127 of them!
Igloo: :P
[18:27]
IglooIt's hard to test stuff as the warrior doesn't have ssh (makes sense)
And debian doesn't support 6.0 anymore in it's repos
[18:28]
Smileyyou could install it and enable it? [18:28]
Kazright [18:28]
Smileythere's only one rsync target atm, i guessing that's still right [18:28]
IglooYeah Smiley but debgs etc [18:28]
Kazpulled, version bumped + put in tracker
oh ffs
right
time to find who to blame
[18:28]
Smileyfor? [18:31]
Kazhttps://s.kurt.gg/20cWA1xw.png [18:31]
Smileylol ;D [18:31]
IglooNot guilty! [18:31]
HCross2I'll kill the Electroneum miner in a bit on master if we need CPU [18:32]
SmileyHCross2: are you seeing it working?
is anyone running a warrior instance? I can go fire a really old one up to test...
[18:32]
IglooI'm running one Smiley
For testing
[18:32]
Smileyis it working?
still the only person i see is JSharp who may of chose the project manually, tht's all.
[18:32]
Kaznobody's returned anything with the new code [18:33]
Smileybecause borked? :/
we broke everyones warriors \o/
[18:34]
HCross2Oops [18:34]
IglooYeah, It'll be broke due to the issue in pipeline
So, That needs to be fixed.
[18:34]
Smileyah well, do they still have the 24hr reset or whatever? [18:34]
IglooI think what HCross2 updated [18:34]
SmileyHCross2: ¬_¬ [18:35]
HCross2Did I commit something [18:35]
Smileyfalacies! [18:35]
HCross2I honestly don't remember [18:35]
IglooIn good news, The warrior-install seems to work OK [18:36]
Kazyeah
what the hell actually broke though
[18:36]
IglooYeah you moved dedupe to a new file HCross2 [18:36]
Kaznothing to do with GetItemFromTracker broke? [18:36]
HCross2Ohh.. I didn't commit that, I just asked for another opinion [18:36]
Kazoh hang on
I'm just stupid, I ran python pipeline.py rather than run-pipeline pipeline.py
(but none of you realised either, anyway!)
[18:36]
IglooOn a warrior running pipeline I get a warcio error [18:38]
Kazreboot
it'll work the second time
[18:38]
HCross2Oooh.. does warcio get setup in the warrior? [18:38]
Kazwarcio gets pulled in as usual, it just only works on the second run [18:39]
HCross2Hmm. Won't that affect the warriors [18:39]
Kazyes
but hang on, genius plan incoming
if we change default project to something else, then back to newsgrabber
[18:39]
Smileythey onl;y update on a new item, don't they? [18:40]
Igloolol [18:40]
Kazuhm [18:40]
HCross2Just move the version number up one? [18:40]
IglooI've just got a symlink to /data/data/data
Someone borked it.
[18:41]
Smileyit's been a loooooooooong time since i did anything. [18:41]
Kazis anyone here running a warrior atm? (and what is it doing?) [18:41]
Smileyno but i can do. [18:41]
IglooI've got one running selected & that works (It hasn't done a code update I don't think)
I have another which i'm prodding to make it work
[18:42]
KazHecatz has updated + looks to be working
so we just need to fix the warriors, working around the fact we may have buggered them
[18:43]
Hecatzyeah, command line though but it's working fine [18:44]
Kazright, spinning a warrior up [18:45]
IglooI'm working on testing the warror [18:45]
Smileywhat port is the warrior on again? http? [18:45]
Kaz8001? [18:46]
Smileyk
hmm not starting headless
i've gotta go put kids to bed
Sl 18:44 0:32 /usr/lib64/virtualbox/VBoxHeadless -s archiveteam-warrior-2
seems to be running
[18:46]
Igloohttps://imgur.com/a/d3kLr
Another typo on that line Kaz needs a space before the ]
I got dinner, I'll be back to fix it later
I've fixed it locally and it's showing working but not install correctly & ingoring
So not sure
[18:49]
Kazupdated [18:53]
........ (idle for 38mn)
https://s.kurt.gg/20dgpamC.png
removed exit 1 from the script
[19:31]
***BnARobin_ has quit IRC (Remote host closed the connection)
BnARobin has joined #newsgrabber
[19:32]
KazIgloo: I fixed it all
I'll bump version later, games first
need to test if it works after a reboot, as it might not
[19:33]
Smileybtw do we have a upload limit in place yet?
speed limiter i mean, as it's easy with rsync
[19:35]
HCross2Nope [19:40]
Smileyworks for my warrior
k i might look at writing a patch for it :O
[19:53]
IglooDo we need it? [19:57]
SmileyI do [19:57]
IglooLooks like it's all working Kaz
Cheers for fixing :D
Smiley: create a patch :-) Though find a way to integrate with the warrior control panel plz
Disabled by default with a button to permit it
[20:03]
Smileyyeah
thats exactly what i want
I only haver 3Mbit upload
[20:06]
............ (idle for 58mn)
IglooUh, The other day there was a spot.im issue
anyone know what it was?
Oh yeah, I remember
[21:04]
Does this project not create a ignore file for wpull? [21:14]
....... (idle for 34mn)
Atomis it normal for the warrior to DNS query newsgrabberdedupe.b-cdn.net tens of times per second? seems to happen on the dedupe step of an item [21:48]
IglooIt's the CDN used for the dedupe process
The TTL is 10 seconds on the entry
Which isn't going to help
Basically, Yeah, It's going to do that. It might be possible to update it to cache it for a period of time
[21:51]
Atomalright, ty [21:54]
IglooBecause it's a CDN, It's going to have a low TTL [21:54]
Atomis it some firewall avoiding cleverness, or does it really need to perform a lookup for every file in the collection? [21:56]
IglooNo, Reading dedupe.py it appears to call it per entry
Might be optomisable but need to check that
[21:56]
Atomalright, thanks for the info [21:57]
.... (idle for 18mn)
Kazuh
right
I think I need to update the version in tracker
Hecatz / odemg / BnAboyZ / Igloo will need to update workers etc
[22:15]
BnAboyZKaz: will do! [22:18]
IglooShould auto update
Hm, maybe not script versions
[22:19]
Kazwarriors will, scripts wont [22:25]
JAAIgloo: Plain wpull doesn't have an ignore file, only the --accept-regex and --reject-regex options (which can only be used once each, otherwise the later occurrence overwrites the previous patterns rather than ORing them). [22:35]
...... (idle for 28mn)
BnAboyZlooks like there is alot of infinite looping going on: https://launcher.spot.im/spot/launcher.spot.im/launcher.spot.im/www.spot.im/launcher/modules/launcher/modules/launcher/modules/launcher/launcher.spot.im/bundle.js\ [23:03]
JAAYeah, that was mentioned previously. I posted an ignore pattern we've been using in ArchiveBot on 2017-12-31 22:50:24 UTC. [23:09]

↑back Search ←Prev date Next date→ Show only urls(Click on time to select a line by its url)