[00:37] *** oldcad has quit IRC (Quit: Leaving.) [01:02] *** yipdw changes topic to: Yes, it's stopped, go read a book or something | Latest Warrior: http://archive.org/download/archiveteam-warrior/archiveteam-warrior-v2-20121008.ova [01:12] *** Ctrl-S has quit IRC (Ping timeout: 252 seconds) [01:14] *** Ctrl-S has joined #warrior [03:34] *** trs80 has quit IRC (Ping timeout: 186 seconds) [03:34] *** trs80 has joined #warrior [04:18] ok i'm here [04:18] yipdw, chfoo, ersi: ping [04:18] PONG xmc [04:19] 2/4, progress! [04:19] 50%, almost good enough for quorum [04:19] let's do this thing [04:19] 50% is good enough for quorum in my senate [04:19] good enough for mongodb lel [04:19] ha [04:20] we use mongodb don't we [04:20] so you turned off replication when? [04:20] I didn't, I thought you wanted to [04:20] ok [04:22] xmc: replication off [04:22] is everything stoppered at 0? [04:22] checking [04:22] let me turn off the urlteam tracker on the new host [04:23] lastfm was set to 5 [04:23] ok [04:23] I'll put that at zero [04:24] ok, one last replication [04:25] oh, i need to update warriorhq, right? [04:25] done [04:25] done the replication? [04:25] oh, yeah [04:25] replication yes [04:25] * xmc investigates warriorhq [04:25] I'll leave the link open a bit longer so some clients can finish checking in [04:26] $ talk chfoo [04:26] -bash: talk: command not found [04:26] :( [04:26] * xmc install [04:27] actually, while we're doing this we might as well also do an apt-get dist-upgrade and reboot [04:27] on zloty [04:27] ok [04:27] ylast update to warriorhq is from may 1 [04:27] just git pull 'd [04:27] is this the correct place https://github.com/ArchiveTeam/warrior-hq [04:27] it should be [04:28] wow, May 1, 2013 [04:28] ok, then it's most recent [04:28] the projects dir in the warrior-hq isn't the latest though [04:28] * xmc apt-get 'ing [04:28] chfoo: can you update that? [04:29] i just tar'ed it from the old tracker and copied into it [04:29] wellllll ok [04:29] is it not in git? [04:29] projects is a private git repo though [04:29] ah [04:30] apt-get upgrade'ing stuff, then i'll reboot the box, does this sound good? [04:30] and we'll see that everything comes up nicely [04:30] which it will (fingers crossed) [04:30] sounds ok to me [04:30] when will the dns changes be made? [04:30] whenever i can raise sketch again [04:31] just texted him [04:31] ¯\_(ツ)_/¯ [04:31] argh damnit xfire is still active [04:31] how the fuck [04:31] there are jobs out? [04:32] area51 is still running i assume [04:32] maybe [04:32] wait shit hmmmm redis is replicating through an ssh tunnel ok this is fine [04:32] actually, no, it isn't [04:32] the replicated data is stuff like the req/min counter [04:32] well the ssh tunnel is being held open by the replication stream [04:33] I'll shutit down [04:33] tunnel is in my screen session [04:33] replication offline [04:33] Connection to tracker.archiveteam.org closed. [04:33] Transferred: sent 59292056, received 201314967240 bytes, in 8644754.1 seconds [04:33] Bytes per second: sent 6.9, received 23287.5 [04:33] debug1: Exit status 0 [04:33] wow 201 GB [04:33] 21:33:33 SketchCow | tracker Old record type was: A [04:33] 21:33:33 SketchCow | New record type is: CNAME [04:34] 21:33:33 SketchCow | Old value: 176.58.114.30 [04:34] 21:33:34 SketchCow | New Value: xn--zoty-01a.xrtc.net [04:34] that replication link was open for MONTHS [04:34] ok the dns is correctly propagated, at least on my machine [04:35] it'll take around 3 hours if my previous dig was accurate [04:35] yep [04:35] ditto [04:35] is warriorhq a cname too? [04:35] ugh, that's an A also [04:36] i'm going to reboot the machine, any reason not to? [04:36] chfoo: yipdw: [04:36] none I know of [04:36] it's ok for me [04:37] k [04:37] bong [04:38] and it's back up [04:38] heh, I heard that in my head [04:39] mmm so it looks like httpd isn't up [04:39] everything looks okay [04:39] Redis is eating 783 MB, of course [04:39] heh [04:40] [duncan@xn--zoty-01a:21:40 0 ~]$ nc localhost 80 [04:40] localhost [127.0.0.1] 80 (http) : Connection refused [04:40] whyyyy [04:40] I don't even know what httpd is on there [04:40] I don't see nginx or apache [04:40] nginx [04:40] huh [04:41] $ sudo service nginx start [04:41] sudo: unable to resolve host xn--zoty-01a [04:41] /lib/systemd/system/nginx-tracker.service [04:41] oh! [04:41] ok [04:41] oh wait [04:41] nginx-tracker, not nginx [04:42] why didn't that start on boot though [04:42] how do you debug systemd [04:42] systemctl [04:43] thank you einstein [04:43] urlteam tracker didn't start up either though i copied the config file from nginx [04:43] it's listed as enabled and active, so that's odd [04:43] it's up now [04:43] but it wasn't when it booted [04:44] I guess we can look in the logs [04:44] there are logs? i thought people hated systemd becuase no logs [04:44] journalctl -b [04:44] :P [04:45] not much in there though [04:45] yeah. [04:45] thx aaaaaaaaa [04:46] you can also do journalctl -u unitname [04:46] oh [04:46] $ sudo systemctl enable nginx-tracker.service [04:46] The unit files have no [Install] section. They are not meant to be enabled [04:46] using systemctl. [04:46] NP, had to read one to many docs to find out how to use it [04:46] well that's good [04:46] I guess you need to declare the proper dependency [04:47] *** aaaaaaaaa has quit IRC (Leaving) [04:47] ah, i should do that then [04:48] i'm going to go with WantedBy=multi-user.target [04:48] for now [04:49] either of you mind if i reboot again, yipdw chfoo? [04:49] nope [04:49] nodejs-tracker will need that line also [04:49] ah good point [04:49] the tinytown needs it too [04:51] added [04:51] oh haha [04:51] ? [04:51] the nodejs component is running in debug mode and all of its output goes to syslog [04:51] ooop [04:51] how do we fix that :P [04:51] run with a higher loglevel I guess [04:52] right yeah [04:52] or we can delete all the console.log statements in server.js [04:52] let's do that after this systemd stuff works [04:53] reboot whenever [04:53] where is the directory with the nodejs component [04:53] /home/tracker/broadcaster [04:53] whatever, rebooting [04:54] i can't find the config option in that directory [04:54] there may not be one set, or there may not be one at all [04:55] yeah, looks like there's nothing set [04:55] there are two console.log statements in redisHandle that look suspicious [04:55] let's get some traffic to the websocket and see if removing those makes a difference [04:56] is this a tomorrow project or a this hour project? [04:56] hmm [04:57] it's a "delete lines, restart server and see what happens" [04:57] i would just redirect to /dev/null [04:57] that also works [04:57] hahaha [04:58] hmm [04:58] another important component of this migration is turning the old tracker off and seeing that things don't break [04:58] WantedBy didn't seem to start stuff up? [04:58] hrm [04:58] i know nothing about systemd [05:00] one sec [05:02] oh wait [05:02] ? [05:02] there's /lib/systemd/system and /etc/systemd/system and /usr/lib/systemd [05:02] I forgot which one unit files go into [05:03] there are a bunch in /lib/systemd/system including nodejs-tracker and nginx, and tinytown in /etc/systemd/system [05:03] OH [05:03] right [05:03] wants are specified in the filesystem [05:03] the WantedBy= tells systemctl what to do [05:04] let's move all our units to /etc/systemd/system just because I get the feeling that's where our shit should go [05:04] that's probably a good idea [05:05] done [05:05] services now registered [05:05] check in /etc/systemd/system/multi-user.target.wants [05:05] chfoo: are you able to give me a list of out item and out that are still to be done from area51? [05:05] \o/ the webserver is up now [05:05] go ahead and reboot just to make sure [05:06] rebooting [05:06] i assumed you did, because my ssh timed out [05:06] mine timed out too [05:06] huh [05:06] well, passenger error [05:07] SOMETHING is up [05:07] there it's all up now [05:07] \o/ automated correctly! [05:07] stage 2, reproducible configuration [05:07] :P [05:07] the warrior frontpage isn't showing a list of projects [05:08] but this looks correct http://zloty.xrtc.net/area51/ [05:09] I see the projects on the front page [05:11] chfoo: tinytown's also logging quite a lot of data to syslog; are all the logs required? [05:11] I guess it's not too big a deal just yet [05:13] hm, i still don't see the projects [05:13] weird [05:13] I redirected nodejs-tracker's stdout to /dev/null and things seem better [05:13] if you see them, that's good enough for me [05:13] where are you looking? [05:13] if I go to http://xn--zoty-01a.xrtc.net/ I see the current projects [05:13] yeah i don't see them there [05:13] whatever [05:14] weird [05:14] oh [05:14] that goes through warriorhq.archiveteam.org [05:14] which is NXDOMAIN in my local resolver for the moment [05:14] oh uh hm [05:14] xn--zoty-01a.xrtc.net.archiveteam.org. [05:14] that's not the right cname! [05:15] silly jason [05:15] heh, no trailing . I guess [05:15] yep [05:16] well he'll fix that eventually [05:17] is anything else broken? [05:17] not that I know of, we'll know when more people hit it [05:17] arkiver: not right now, i have to sleep soon [05:17] is it open to handing out jobs yet? [05:17] sure [05:17] yipdw: the tracker logs into it's own files so i guess redirect the output to /dev/null too [05:18] oh [05:19] done [05:19] *** yipdw changes topic to: Open for business if your DNS provider is | Latest Warrior: http://archive.org/download/archiveteam-warrior/archiveteam-warrior-v2-20121008.ova [05:21] arkiver: ping [05:21] arkiver: does everything look good to you [05:22] http://zloty.xrtc.net/area51/admin/ will definitely go to the right place [05:26] chfoo, yipdw: will things explode if I turn off shilling, so we can be sure we're not using it [05:26] well clients that haven't resolved the new name will be confused [05:26] and warriorhq won't resolve until that CNAME gets fixed [05:26] right but they shouldn't be working on the old database anyway [05:26] it's off [05:26] other than that, no [05:27] warriorhq goes to lala land right now :P [05:27] you may want to shut off the httpd first, do a backup of /home and /var [05:27] and then turn it off [05:27] I can't remember if we did anything live [05:27] good point [05:27] i'm not going to kill it for a day or so at least [05:27] just power it off [05:27] ah ok that's fine then [05:27] cool [05:28] yeah i'm going to 'tar /' to a machine i have at home, in case we need it next month [05:30] i want to figure out why the munin graphs aren't graphing first [05:30] http://zeppelin.xrtc.net/corp.xrtc.net/zloty.corp.xrtc.net/index.html [05:31] also i want to watch this fall off a cliff naturally http://zeppelin.xrtc.net/corp.xrtc.net/shilling.corp.xrtc.net/warriors.html [05:32] hrmph [05:32] there's no reason it shouldn't work [05:32] are you sure they aren't working? [05:32] there might just be no Warrior activity [05:33] well the graphs don't render [05:33] oh wait, the by-project graphs [05:33] http://zeppelin.xrtc.net/corp.xrtc.net/zloty.corp.xrtc.net/project_bytes.html [05:33] ok [05:33] yeah [05:34] oh that'll do it [05:34] 2015/07/28 22:30:30 [RRD ERROR] Unable to graph /var/cache/munin/www/corp.xrtc.net/shilling.corp.xrtc.net/project_bytes-week.png : invalid rpn expression in: google-helpouts,8,* [05:35] psh 8 * google-helpouts is defined [05:35] it sees the - and is like HOW DO I SUBTRACT [05:35] SUBTRACT WHAT [05:35] i bet [05:40] $ redis-cli --raw -n 1 keys 'warriorhq:instances:*' [05:40] $ [05:40] * xmc shrug [05:40] how does this work [05:41] is there a thing i need to run [05:41] there should be things contacting it [05:41] looks like items can be received from the tracker, but that tracker then doesn't list the items as out [05:42] yeah hm [05:42] warriorhq is busted [05:42] OH IS THAT BECAUSE OF THE DNS THING [05:42] SketchCow is not in this channel [05:42] i know [05:44] you did pause area51 before the update? [05:44] I see from logs from another user he is receiving items, but when I try I can't receive items [05:45] chfoo: ok. have good night! [05:54] warriorhq.archiveteam.org doesn't need to resolve for items to be marked as out [05:54] weird [05:55] all projects are still at 0 req/min [05:55] so nobody should be getting anything [08:00] *** nmnn has joined #warrior [08:09] *** toad2 has joined #warrior [08:11] *** toad1 has quit IRC (Read error: Operation timed out) [08:14] *** nmnn has quit IRC (Ping timeout: 483 seconds) [11:18] *** nmnn has joined #warrior [12:46] *** nmnn has quit IRC (Ping timeout: 483 seconds) [12:58] *** nmnn has joined #warrior [13:34] *** nmnn has quit IRC (Ping timeout: 483 seconds) [13:44] *** nmnn has joined #warrior [14:04] *** Fletcher has quit IRC (Ping timeout: 252 seconds) [14:05] *** midas has quit IRC (Read error: Operation timed out) [14:10] *** midas has joined #warrior [14:11] *** svchfoo3 sets mode: +o midas [15:02] *** zyphlar has quit IRC (Write error: Broken pipe) [15:19] *** oldcad has joined #warrior [15:19] *** Fletcher has joined #warrior [15:36] *** yipdw changes topic to: Thanks for NOT asking about upload problems | Latest Warrior: http://archive.org/download/archiveteam-warrior/archiveteam-warrior-v2-20121008.ova [16:01] ssl config on nginx is renabled and the live updates setting was fixed [16:13] *** BuildTheR has quit IRC (Remote host closed the connection) [16:25] *** zyphlar has joined #warrior [16:49] *** BuildTheR has joined #warrior [17:11] *** aaaaaaaaa has joined #warrior [17:14] sweet [17:18] back to the old topic. Everything transfer ok? [17:19] looks like you guys had quite the hour after I left [17:27] it seems to have transferred ok [17:28] the graph still looks weird and i'll look at that tonight, in about 10 hours [17:36] *** bsmith096 has quit IRC (Ping timeout: 240 seconds) [18:00] *** nmnn has quit IRC (Quit: Ex-Chat) [18:02] ain't no party like an admin party [19:23] *** oldcad has quit IRC (Quit: Leaving.) [21:09] *** brayden_ has quit IRC (Read error: Operation timed out) [21:48] *** Start has quit IRC (Quit: Disconnected.)