#warrior 2015-07-29,Wed

↑back Search

Time Nickname Message
00:37 🔗 oldcad has quit IRC (Quit: Leaving.)
01:02 🔗 yipdw changes topic to: Yes, it's stopped, go read a book or something | Latest Warrior: http://archive.org/download/archiveteam-warrior/archiveteam-warrior-v2-20121008.ova
01:12 🔗 Ctrl-S has quit IRC (Ping timeout: 252 seconds)
01:14 🔗 Ctrl-S has joined #warrior
03:34 🔗 trs80 has quit IRC (Ping timeout: 186 seconds)
03:34 🔗 trs80 has joined #warrior
04:18 🔗 xmc ok i'm here
04:18 🔗 xmc yipdw, chfoo, ersi: ping
04:18 🔗 yipdw PONG xmc
04:19 🔗 xmc 2/4, progress!
04:19 🔗 yipdw 50%, almost good enough for quorum
04:19 🔗 xmc let's do this thing
04:19 🔗 xmc 50% is good enough for quorum in my senate
04:19 🔗 yipdw good enough for mongodb lel
04:19 🔗 xmc ha
04:20 🔗 xmc we use mongodb don't we
04:20 🔗 xmc so you turned off replication when?
04:20 🔗 yipdw I didn't, I thought you wanted to
04:20 🔗 xmc ok
04:22 🔗 yipdw xmc: replication off
04:22 🔗 xmc is everything stoppered at 0?
04:22 🔗 yipdw checking
04:22 🔗 chfoo let me turn off the urlteam tracker on the new host
04:23 🔗 yipdw lastfm was set to 5
04:23 🔗 xmc ok
04:23 🔗 yipdw I'll put that at zero
04:24 🔗 yipdw ok, one last replication
04:25 🔗 xmc oh, i need to update warriorhq, right?
04:25 🔗 yipdw done
04:25 🔗 xmc done the replication?
04:25 🔗 yipdw oh, yeah
04:25 🔗 yipdw replication yes
04:25 🔗 * xmc investigates warriorhq
04:25 🔗 yipdw I'll leave the link open a bit longer so some clients can finish checking in
04:26 🔗 yipdw $ talk chfoo
04:26 🔗 yipdw -bash: talk: command not found
04:26 🔗 yipdw :(
04:26 🔗 * xmc install
04:27 🔗 yipdw actually, while we're doing this we might as well also do an apt-get dist-upgrade and reboot
04:27 🔗 yipdw on zloty
04:27 🔗 xmc ok
04:27 🔗 xmc ylast update to warriorhq is from may 1
04:27 🔗 xmc just git pull 'd
04:27 🔗 xmc is this the correct place https://github.com/ArchiveTeam/warrior-hq
04:27 🔗 chfoo it should be
04:28 🔗 yipdw wow, May 1, 2013
04:28 🔗 xmc ok, then it's most recent
04:28 🔗 chfoo the projects dir in the warrior-hq isn't the latest though
04:28 🔗 * xmc apt-get 'ing
04:28 🔗 xmc chfoo: can you update that?
04:29 🔗 chfoo i just tar'ed it from the old tracker and copied into it
04:29 🔗 xmc wellllll ok
04:29 🔗 xmc is it not in git?
04:29 🔗 chfoo projects is a private git repo though
04:29 🔗 xmc ah
04:30 🔗 xmc apt-get upgrade'ing stuff, then i'll reboot the box, does this sound good?
04:30 🔗 xmc and we'll see that everything comes up nicely
04:30 🔗 xmc which it will (fingers crossed)
04:30 🔗 chfoo sounds ok to me
04:30 🔗 chfoo when will the dns changes be made?
04:30 🔗 xmc whenever i can raise sketch again
04:31 🔗 xmc just texted him
04:31 🔗 xmc ¯\_(ツ)_/¯
04:31 🔗 yipdw argh damnit xfire is still active
04:31 🔗 yipdw how the fuck
04:31 🔗 xmc there are jobs out?
04:32 🔗 chfoo area51 is still running i assume
04:32 🔗 yipdw maybe
04:32 🔗 xmc wait shit hmmmm redis is replicating through an ssh tunnel ok this is fine
04:32 🔗 yipdw actually, no, it isn't
04:32 🔗 yipdw the replicated data is stuff like the req/min counter
04:32 🔗 xmc well the ssh tunnel is being held open by the replication stream
04:33 🔗 yipdw I'll shutit down
04:33 🔗 xmc tunnel is in my screen session
04:33 🔗 yipdw replication offline
04:33 🔗 xmc Connection to tracker.archiveteam.org closed.
04:33 🔗 xmc Transferred: sent 59292056, received 201314967240 bytes, in 8644754.1 seconds
04:33 🔗 xmc Bytes per second: sent 6.9, received 23287.5
04:33 🔗 xmc debug1: Exit status 0
04:33 🔗 yipdw wow 201 GB
04:33 🔗 xmc 21:33:33 SketchCow | tracker Old record type was: A
04:33 🔗 xmc 21:33:33 SketchCow | New record type is: CNAME
04:34 🔗 xmc 21:33:33 SketchCow | Old value: 176.58.114.30
04:34 🔗 xmc 21:33:34 SketchCow | New Value: xn--zoty-01a.xrtc.net
04:34 🔗 xmc that replication link was open for MONTHS
04:34 🔗 xmc ok the dns is correctly propagated, at least on my machine
04:35 🔗 yipdw it'll take around 3 hours if my previous dig was accurate
04:35 🔗 xmc yep
04:35 🔗 xmc ditto
04:35 🔗 chfoo is warriorhq a cname too?
04:35 🔗 xmc ugh, that's an A also
04:36 🔗 xmc i'm going to reboot the machine, any reason not to?
04:36 🔗 xmc chfoo: yipdw:
04:36 🔗 yipdw none I know of
04:36 🔗 chfoo it's ok for me
04:37 🔗 xmc k
04:37 🔗 xmc bong
04:38 🔗 xmc and it's back up
04:38 🔗 aaaaaaaaa heh, I heard that in my head
04:39 🔗 xmc mmm so it looks like httpd isn't up
04:39 🔗 yipdw everything looks okay
04:39 🔗 yipdw Redis is eating 783 MB, of course
04:39 🔗 xmc heh
04:40 🔗 xmc [duncan@xn--zoty-01a:21:40 0 ~]$ nc localhost 80
04:40 🔗 xmc localhost [127.0.0.1] 80 (http) : Connection refused
04:40 🔗 xmc whyyyy
04:40 🔗 yipdw I don't even know what httpd is on there
04:40 🔗 yipdw I don't see nginx or apache
04:40 🔗 xmc nginx
04:40 🔗 yipdw huh
04:41 🔗 yipdw $ sudo service nginx start
04:41 🔗 yipdw sudo: unable to resolve host xn--zoty-01a
04:41 🔗 xmc /lib/systemd/system/nginx-tracker.service
04:41 🔗 xmc oh!
04:41 🔗 xmc ok
04:41 🔗 yipdw oh wait
04:41 🔗 yipdw nginx-tracker, not nginx
04:42 🔗 xmc why didn't that start on boot though
04:42 🔗 xmc how do you debug systemd
04:42 🔗 yipdw systemctl
04:43 🔗 xmc thank you einstein
04:43 🔗 chfoo urlteam tracker didn't start up either though i copied the config file from nginx
04:43 🔗 yipdw it's listed as enabled and active, so that's odd
04:43 🔗 xmc it's up now
04:43 🔗 xmc but it wasn't when it booted
04:44 🔗 yipdw I guess we can look in the logs
04:44 🔗 xmc there are logs? i thought people hated systemd becuase no logs
04:44 🔗 aaaaaaaaa journalctl -b
04:44 🔗 xmc :P
04:45 🔗 yipdw not much in there though
04:45 🔗 xmc yeah.
04:45 🔗 xmc thx aaaaaaaaa
04:46 🔗 aaaaaaaaa you can also do journalctl -u unitname
04:46 🔗 yipdw oh
04:46 🔗 yipdw $ sudo systemctl enable nginx-tracker.service
04:46 🔗 yipdw The unit files have no [Install] section. They are not meant to be enabled
04:46 🔗 yipdw using systemctl.
04:46 🔗 aaaaaaaaa NP, had to read one to many docs to find out how to use it
04:46 🔗 yipdw well that's good
04:46 🔗 yipdw I guess you need to declare the proper dependency
04:47 🔗 aaaaaaaaa has quit IRC (Leaving)
04:47 🔗 xmc ah, i should do that then
04:48 🔗 xmc i'm going to go with WantedBy=multi-user.target
04:48 🔗 xmc for now
04:49 🔗 xmc either of you mind if i reboot again, yipdw chfoo?
04:49 🔗 yipdw nope
04:49 🔗 yipdw nodejs-tracker will need that line also
04:49 🔗 xmc ah good point
04:49 🔗 chfoo the tinytown needs it too
04:51 🔗 xmc added
04:51 🔗 yipdw oh haha
04:51 🔗 xmc ?
04:51 🔗 yipdw the nodejs component is running in debug mode and all of its output goes to syslog
04:51 🔗 xmc ooop
04:51 🔗 xmc how do we fix that :P
04:51 🔗 yipdw run with a higher loglevel I guess
04:52 🔗 xmc right yeah
04:52 🔗 yipdw or we can delete all the console.log statements in server.js
04:52 🔗 yipdw let's do that after this systemd stuff works
04:53 🔗 yipdw reboot whenever
04:53 🔗 xmc where is the directory with the nodejs component
04:53 🔗 yipdw /home/tracker/broadcaster
04:53 🔗 xmc whatever, rebooting
04:54 🔗 xmc i can't find the config option in that directory
04:54 🔗 yipdw there may not be one set, or there may not be one at all
04:55 🔗 xmc yeah, looks like there's nothing set
04:55 🔗 yipdw there are two console.log statements in redisHandle that look suspicious
04:55 🔗 yipdw let's get some traffic to the websocket and see if removing those makes a difference
04:56 🔗 xmc is this a tomorrow project or a this hour project?
04:56 🔗 yipdw hmm
04:57 🔗 yipdw it's a "delete lines, restart server and see what happens"
04:57 🔗 chfoo i would just redirect to /dev/null
04:57 🔗 yipdw that also works
04:57 🔗 xmc hahaha
04:58 🔗 yipdw hmm
04:58 🔗 xmc another important component of this migration is turning the old tracker off and seeing that things don't break
04:58 🔗 yipdw WantedBy didn't seem to start stuff up?
04:58 🔗 xmc hrm
04:58 🔗 xmc i know nothing about systemd
05:00 🔗 yipdw one sec
05:02 🔗 yipdw oh wait
05:02 🔗 xmc ?
05:02 🔗 yipdw there's /lib/systemd/system and /etc/systemd/system and /usr/lib/systemd
05:02 🔗 yipdw I forgot which one unit files go into
05:03 🔗 xmc there are a bunch in /lib/systemd/system including nodejs-tracker and nginx, and tinytown in /etc/systemd/system
05:03 🔗 yipdw OH
05:03 🔗 yipdw right
05:03 🔗 yipdw wants are specified in the filesystem
05:03 🔗 yipdw the WantedBy= tells systemctl what to do
05:04 🔗 yipdw let's move all our units to /etc/systemd/system just because I get the feeling that's where our shit should go
05:04 🔗 xmc that's probably a good idea
05:05 🔗 yipdw done
05:05 🔗 yipdw services now registered
05:05 🔗 yipdw check in /etc/systemd/system/multi-user.target.wants
05:05 🔗 arkiver chfoo: are you able to give me a list of out item and out that are still to be done from area51?
05:05 🔗 xmc \o/ the webserver is up now
05:05 🔗 yipdw go ahead and reboot just to make sure
05:06 🔗 xmc rebooting
05:06 🔗 xmc i assumed you did, because my ssh timed out
05:06 🔗 yipdw mine timed out too
05:06 🔗 xmc huh
05:06 🔗 xmc well, passenger error
05:07 🔗 xmc SOMETHING is up
05:07 🔗 xmc there it's all up now
05:07 🔗 xmc \o/ automated correctly!
05:07 🔗 yipdw stage 2, reproducible configuration
05:07 🔗 xmc :P
05:07 🔗 xmc the warrior frontpage isn't showing a list of projects
05:08 🔗 xmc but this looks correct http://zloty.xrtc.net/area51/
05:09 🔗 yipdw I see the projects on the front page
05:11 🔗 yipdw chfoo: tinytown's also logging quite a lot of data to syslog; are all the logs required?
05:11 🔗 yipdw I guess it's not too big a deal just yet
05:13 🔗 xmc hm, i still don't see the projects
05:13 🔗 xmc weird
05:13 🔗 yipdw I redirected nodejs-tracker's stdout to /dev/null and things seem better
05:13 🔗 xmc if you see them, that's good enough for me
05:13 🔗 yipdw where are you looking?
05:13 🔗 yipdw if I go to http://xn--zoty-01a.xrtc.net/ I see the current projects
05:13 🔗 xmc yeah i don't see them there
05:13 🔗 xmc whatever
05:14 🔗 yipdw weird
05:14 🔗 xmc oh
05:14 🔗 xmc that goes through warriorhq.archiveteam.org
05:14 🔗 xmc which is NXDOMAIN in my local resolver for the moment
05:14 🔗 xmc oh uh hm
05:14 🔗 xmc xn--zoty-01a.xrtc.net.archiveteam.org.
05:14 🔗 xmc that's not the right cname!
05:15 🔗 xmc silly jason
05:15 🔗 yipdw heh, no trailing . I guess
05:15 🔗 xmc yep
05:16 🔗 xmc well he'll fix that eventually
05:17 🔗 xmc is anything else broken?
05:17 🔗 yipdw not that I know of, we'll know when more people hit it
05:17 🔗 chfoo arkiver: not right now, i have to sleep soon
05:17 🔗 xmc is it open to handing out jobs yet?
05:17 🔗 yipdw sure
05:17 🔗 chfoo yipdw: the tracker logs into it's own files so i guess redirect the output to /dev/null too
05:18 🔗 yipdw oh
05:19 🔗 yipdw done
05:19 🔗 yipdw changes topic to: Open for business if your DNS provider is | Latest Warrior: http://archive.org/download/archiveteam-warrior/archiveteam-warrior-v2-20121008.ova
05:21 🔗 xmc arkiver: ping
05:21 🔗 xmc arkiver: does everything look good to you
05:22 🔗 xmc http://zloty.xrtc.net/area51/admin/ will definitely go to the right place
05:26 🔗 xmc chfoo, yipdw: will things explode if I turn off shilling, so we can be sure we're not using it
05:26 🔗 yipdw well clients that haven't resolved the new name will be confused
05:26 🔗 yipdw and warriorhq won't resolve until that CNAME gets fixed
05:26 🔗 xmc right but they shouldn't be working on the old database anyway
05:26 🔗 xmc it's off
05:26 🔗 yipdw other than that, no
05:27 🔗 xmc warriorhq goes to lala land right now :P
05:27 🔗 yipdw you may want to shut off the httpd first, do a backup of /home and /var
05:27 🔗 yipdw and then turn it off
05:27 🔗 yipdw I can't remember if we did anything live
05:27 🔗 xmc good point
05:27 🔗 xmc i'm not going to kill it for a day or so at least
05:27 🔗 xmc just power it off
05:27 🔗 yipdw ah ok that's fine then
05:27 🔗 xmc cool
05:28 🔗 xmc yeah i'm going to 'tar /' to a machine i have at home, in case we need it next month
05:30 🔗 xmc i want to figure out why the munin graphs aren't graphing first
05:30 🔗 xmc http://zeppelin.xrtc.net/corp.xrtc.net/zloty.corp.xrtc.net/index.html
05:31 🔗 xmc also i want to watch this fall off a cliff naturally http://zeppelin.xrtc.net/corp.xrtc.net/shilling.corp.xrtc.net/warriors.html
05:32 🔗 xmc hrmph
05:32 🔗 xmc there's no reason it shouldn't work
05:32 🔗 yipdw are you sure they aren't working?
05:32 🔗 yipdw there might just be no Warrior activity
05:33 🔗 xmc well the graphs don't render
05:33 🔗 yipdw oh wait, the by-project graphs
05:33 🔗 xmc http://zeppelin.xrtc.net/corp.xrtc.net/zloty.corp.xrtc.net/project_bytes.html
05:33 🔗 yipdw ok
05:33 🔗 xmc yeah
05:34 🔗 xmc oh that'll do it
05:34 🔗 xmc 2015/07/28 22:30:30 [RRD ERROR] Unable to graph /var/cache/munin/www/corp.xrtc.net/shilling.corp.xrtc.net/project_bytes-week.png : invalid rpn expression in: google-helpouts,8,*
05:35 🔗 yipdw psh 8 * google-helpouts is defined
05:35 🔗 xmc it sees the - and is like HOW DO I SUBTRACT
05:35 🔗 xmc SUBTRACT WHAT
05:35 🔗 xmc i bet
05:40 🔗 xmc $ redis-cli --raw -n 1 keys 'warriorhq:instances:*'
05:40 🔗 xmc $
05:40 🔗 * xmc shrug
05:40 🔗 xmc how does this work
05:41 🔗 xmc is there a thing i need to run
05:41 🔗 xmc there should be things contacting it
05:41 🔗 arkiver looks like items can be received from the tracker, but that tracker then doesn't list the items as out
05:42 🔗 xmc yeah hm
05:42 🔗 xmc warriorhq is busted
05:42 🔗 xmc OH IS THAT BECAUSE OF THE DNS THING
05:42 🔗 arkiver SketchCow is not in this channel
05:42 🔗 xmc i know
05:44 🔗 arkiver you did pause area51 before the update?
05:44 🔗 arkiver I see from logs from another user he is receiving items, but when I try I can't receive items
05:45 🔗 arkiver chfoo: ok. have good night!
05:54 🔗 yipdw warriorhq.archiveteam.org doesn't need to resolve for items to be marked as out
05:54 🔗 xmc weird
05:55 🔗 yipdw all projects are still at 0 req/min
05:55 🔗 yipdw so nobody should be getting anything
08:00 🔗 nmnn has joined #warrior
08:09 🔗 toad2 has joined #warrior
08:11 🔗 toad1 has quit IRC (Read error: Operation timed out)
08:14 🔗 nmnn has quit IRC (Ping timeout: 483 seconds)
11:18 🔗 nmnn has joined #warrior
12:46 🔗 nmnn has quit IRC (Ping timeout: 483 seconds)
12:58 🔗 nmnn has joined #warrior
13:34 🔗 nmnn has quit IRC (Ping timeout: 483 seconds)
13:44 🔗 nmnn has joined #warrior
14:04 🔗 Fletcher has quit IRC (Ping timeout: 252 seconds)
14:05 🔗 midas has quit IRC (Read error: Operation timed out)
14:10 🔗 midas has joined #warrior
14:11 🔗 svchfoo3 sets mode: +o midas
15:02 🔗 zyphlar has quit IRC (Write error: Broken pipe)
15:19 🔗 oldcad has joined #warrior
15:19 🔗 Fletcher has joined #warrior
15:36 🔗 yipdw changes topic to: Thanks for NOT asking about upload problems | Latest Warrior: http://archive.org/download/archiveteam-warrior/archiveteam-warrior-v2-20121008.ova
16:01 🔗 chfoo ssl config on nginx is renabled and the live updates setting was fixed
16:13 🔗 BuildTheR has quit IRC (Remote host closed the connection)
16:25 🔗 zyphlar has joined #warrior
16:49 🔗 BuildTheR has joined #warrior
17:11 🔗 aaaaaaaaa has joined #warrior
17:14 🔗 xmc sweet
17:18 🔗 aaaaaaaaa back to the old topic. Everything transfer ok?
17:19 🔗 aaaaaaaaa looks like you guys had quite the hour after I left
17:27 🔗 xmc it seems to have transferred ok
17:28 🔗 xmc the graph still looks weird and i'll look at that tonight, in about 10 hours
17:36 🔗 bsmith096 has quit IRC (Ping timeout: 240 seconds)
18:00 🔗 nmnn has quit IRC (Quit: Ex-Chat)
18:02 🔗 yipdw ain't no party like an admin party
19:23 🔗 oldcad has quit IRC (Quit: Leaving.)
21:09 🔗 brayden_ has quit IRC (Read error: Operation timed out)
21:48 🔗 Start has quit IRC (Quit: Disconnected.)
irclogger-viewer