[00:22] i'm going to take advantage of the down time to set up log rotation for nginx on the tracker. it grew pretty quicjly [00:34] you may want to remove some of the done items as well, to lighten the memory load. [00:40] nginx doesn't seem to respond to kill -USR1 `cat /var/run/nginx.pid` [00:43] doesn't the graceful exit usually take awhile? [00:44] the docs say it's supposed to reopen the log files but it's still writing to error.log.1 [00:45] does nginx have execute permission on the log directory? [00:45] -rw-r--r-- 1 www-data root 21G Sep 11 20:45 error.log.1 [00:46] drwxr-xr-x 13 root root 4.0K Sep 11 06:25 .. [00:46] i'm guessing the owner needs to be changed [00:48] oops, wrong line. it's: drwxr-xr-x 2 www-data root 4.0K Sep 11 20:35 . [00:54] hmm, maybe chown www-data:www-data? [00:55] still doesn't work [01:02] is that the right pid file? [01:04] pidof nginx gives 15548 15547. cat /var/run/nginx.pid gives 15547 [01:04] The only other thing I can thing of is that nginx won't restart the workers because they are still working. [01:05] you could try to force restart them with a hup and see what happens [01:05] but I don't know if I'd do that on my own server that is doing work [01:06] strange. doing sudo /usr/local/nginx/sbin/nginx -s reopen seems to have worked [01:07] nginx version: nginx/1.2.8 [01:08] that is the same as usr1 [01:08] oh wait, it's still writing to error.log.1 [01:08] wonder if it just took awhile for all the workers to quit [01:08] only the new ones would have rotated, IIRC [01:09] do you have two error logs now? [01:09] or still just the one? [01:09] yes, error.log is 61 bytes whiel error.log.1 is 22488702033 and growing [01:12] HUP seems to have worked [01:13] that forces it to reload the configuration, not just the logs. [01:14] maybe you have to do that the first time and USR1 works on the later ones? [01:15] I can't believe the size of those logs, BTW. Maybe I'm just used to less popular sites [01:17] i'm not sure if i want to investigate too much so i just i set the log rotate config postrotate section to to HUP, USR1, HUP, USR1 [01:18] the error log is just filled with: Uncaught exception in PassengerServer client thread: exception: Cannot read response from backend process: Connection reset by peer (104) [01:21] oh yeah. I don't like passenger ever since the "incident." [01:23] but the tracker itself is nice [01:26] yipdw: i noticed that we're both running log drainer at the same time. [01:40] on further research, it appears to be reasonably safe [02:53] chfoo: oh, feel free to kill one [08:16] drain all the logs [16:05] chfoo: FYI; renaming a file doesn't change what file an application will write to unless it closes and reopens it [16:06] it's tied to the inode, not the path [16:06] so most software will require some kind of hint that it has to reopen the file [16:06] (which I presume is what HUP does) [16:07] i know. usr1 does that, but nginx's workers don't listen to that [16:08] hup causes the tracker to go unavailable for a few minutes [16:08] hup makes it reload all the config files [16:08] logrotate also has copy and truncate but the logs are gigabytes in size so i didn't want to use that [16:09] do you know the lifespan of nginx's workers on your system? [16:10] USR1 makes the new ones use the new logs, but old ones will use the old ones until they term. [16:11] HUP waits for all workers to die, reloads and then creates new ones. [16:11] the config file doesn't specify the lifetime. just max pool is 24 [16:12] maybe you just have long lived ones? [16:46] i guess that might be why [17:35] suggestion: the tracker places the current items per minute at the top. e.g. Verizon tracker | Currently 1400 items per minute | Please lower your concurrency; tracker exceeded C10K [21:05] next person to use C10K in a sentence gets stabbed in the face over the internet [21:06] also wtf libavutil, where is av_frame_alloc [21:07] aaaaaaaaa: there's an issue sorta related to that -> https://github.com/ArchiveTeam/universal-tracker/issues/22 [21:23] Hmm, looked through the bug tracker before I made mine, must have gone right over that one. [21:24] In any case, my apologies for offending your sensibilities; I just copied the text from the current verizon tracker for my suggestion. [21:31] C10K? [21:47] 13seesaw-kit/06development 1468cd3d6 15Christopher Foo: warrior ui: Change broadcast msg indicator to darkgoldenrod... [21:47] 13seesaw-kit/06development 147e51ee8 15Christopher Foo: warrior ui: Decrease header padding. Add .projectBroadcastMessage to styles.... [21:47] [13seesaw-kit] 15chfoo pushed 2 new commits to 06development: 02https://github.com/ArchiveTeam/seesaw-kit/compare/a5ff73dbe94e...7e51ee832123 [21:56] 13seesaw-kit/06development 1448f54e9 15Christopher Foo: web.py: Use MD5 digest as hash for indicator message... [21:56] [13seesaw-kit] 15chfoo pushed 1 new commit to 06development: 02https://github.com/ArchiveTeam/seesaw-kit/commit/48f54e91b43cee3d237615e07ad05f29803624a8 [22:17] 13seesaw-kit/06development 14043681a 15Anthony Eden: Add time indicator to each item. Shows how long it's been running. [22:17] 13seesaw-kit/06development 14c25b066 15Christopher Foo: Merge branch 'temp1' into development... [22:17] [13seesaw-kit] 15chfoo pushed 2 new commits to 06development: 02https://github.com/ArchiveTeam/seesaw-kit/compare/48f54e91b43c...c25b0669dd16 [23:33] 13seesaw-kit/06development 1430456fb 15Christopher Foo: fixup! Merge branch 'temp1' into development... [23:33] 13seesaw-kit/06development 146944392 15Christopher Foo: warrior ui: Remove 'Task Status Summary' h3 into a title attr. Fix align. [23:33] 13seesaw-kit/06development 14c5d0911 15Christopher Foo: warrior ui: Move project name and timer under log. 'Runtime'→'Elapsed'.... [23:33] [13seesaw-kit] 15chfoo pushed 3 new commits to 06development: 02https://github.com/ArchiveTeam/seesaw-kit/compare/c25b0669dd16...c5d0911c27db