| Time |
Nickname |
Message |
|
00:22
๐
|
chfoo |
i'm going to take advantage of the down time to set up log rotation for nginx on the tracker. it grew pretty quicjly |
|
00:34
๐
|
aaaaaaaaa |
you may want to remove some of the done items as well, to lighten the memory load. |
|
00:40
๐
|
chfoo |
nginx doesn't seem to respond to kill -USR1 `cat /var/run/nginx.pid` |
|
00:43
๐
|
aaaaaaaaa |
doesn't the graceful exit usually take awhile? |
|
00:44
๐
|
chfoo |
the docs say it's supposed to reopen the log files but it's still writing to error.log.1 |
|
00:45
๐
|
aaaaaaaaa |
does nginx have execute permission on the log directory? |
|
00:45
๐
|
chfoo |
-rw-r--r-- 1 www-data root 21G Sep 11 20:45 error.log.1 |
|
00:46
๐
|
chfoo |
drwxr-xr-x 13 root root 4.0K Sep 11 06:25 .. |
|
00:46
๐
|
chfoo |
i'm guessing the owner needs to be changed |
|
00:48
๐
|
chfoo |
oops, wrong line. it's: drwxr-xr-x 2 www-data root 4.0K Sep 11 20:35 . |
|
00:54
๐
|
aaaaaaaaa |
hmm, maybe chown www-data:www-data? |
|
00:55
๐
|
chfoo |
still doesn't work |
|
01:02
๐
|
aaaaaaaaa |
is that the right pid file? |
|
01:04
๐
|
chfoo |
pidof nginx gives 15548 15547. cat /var/run/nginx.pid gives 15547 |
|
01:04
๐
|
aaaaaaaaa |
The only other thing I can thing of is that nginx won't restart the workers because they are still working. |
|
01:05
๐
|
aaaaaaaaa |
you could try to force restart them with a hup and see what happens |
|
01:05
๐
|
aaaaaaaaa |
but I don't know if I'd do that on my own server that is doing work |
|
01:06
๐
|
chfoo |
strange. doing sudo /usr/local/nginx/sbin/nginx -s reopen seems to have worked |
|
01:07
๐
|
chfoo |
nginx version: nginx/1.2.8 |
|
01:08
๐
|
aaaaaaaaa |
that is the same as usr1 |
|
01:08
๐
|
chfoo |
oh wait, it's still writing to error.log.1 |
|
01:08
๐
|
aaaaaaaaa |
wonder if it just took awhile for all the workers to quit |
|
01:08
๐
|
aaaaaaaaa |
only the new ones would have rotated, IIRC |
|
01:09
๐
|
aaaaaaaaa |
do you have two error logs now? |
|
01:09
๐
|
aaaaaaaaa |
or still just the one? |
|
01:09
๐
|
chfoo |
yes, error.log is 61 bytes whiel error.log.1 is 22488702033 and growing |
|
01:12
๐
|
chfoo |
HUP seems to have worked |
|
01:13
๐
|
aaaaaaaaa |
that forces it to reload the configuration, not just the logs. |
|
01:14
๐
|
aaaaaaaaa |
maybe you have to do that the first time and USR1 works on the later ones? |
|
01:15
๐
|
aaaaaaaaa |
I can't believe the size of those logs, BTW. Maybe I'm just used to less popular sites |
|
01:17
๐
|
chfoo |
i'm not sure if i want to investigate too much so i just i set the log rotate config postrotate section to to HUP, USR1, HUP, USR1 |
|
01:18
๐
|
chfoo |
the error log is just filled with: Uncaught exception in PassengerServer client thread: exception: Cannot read response from backend process: Connection reset by peer (104) |
|
01:21
๐
|
aaaaaaaaa |
oh yeah. I don't like passenger ever since the "incident." |
|
01:23
๐
|
aaaaaaaaa |
but the tracker itself is nice |
|
01:26
๐
|
chfoo |
yipdw: i noticed that we're both running log drainer at the same time. |
|
01:40
๐
|
chfoo |
on further research, it appears to be reasonably safe |
|
02:53
๐
|
yipdw |
chfoo: oh, feel free to kill one |
|
08:16
๐
|
midas |
drain all the logs |
|
16:05
๐
|
joepie91 |
chfoo: FYI; renaming a file doesn't change what file an application will write to unless it closes and reopens it |
|
16:06
๐
|
joepie91 |
it's tied to the inode, not the path |
|
16:06
๐
|
joepie91 |
so most software will require some kind of hint that it has to reopen the file |
|
16:06
๐
|
joepie91 |
(which I presume is what HUP does) |
|
16:07
๐
|
chfoo |
i know. usr1 does that, but nginx's workers don't listen to that |
|
16:08
๐
|
chfoo |
hup causes the tracker to go unavailable for a few minutes |
|
16:08
๐
|
aaaaaaaaa |
hup makes it reload all the config files |
|
16:08
๐
|
chfoo |
logrotate also has copy and truncate but the logs are gigabytes in size so i didn't want to use that |
|
16:09
๐
|
aaaaaaaaa |
do you know the lifespan of nginx's workers on your system? |
|
16:10
๐
|
aaaaaaaaa |
USR1 makes the new ones use the new logs, but old ones will use the old ones until they term. |
|
16:11
๐
|
aaaaaaaaa |
HUP waits for all workers to die, reloads and then creates new ones. |
|
16:11
๐
|
chfoo |
the config file doesn't specify the lifetime. just max pool is 24 |
|
16:12
๐
|
aaaaaaaaa |
maybe you just have long lived ones? |
|
16:46
๐
|
chfoo |
i guess that might be why |
|
17:35
๐
|
aaaaaaaaa |
suggestion: the tracker places the current items per minute at the top. e.g. Verizon tracker | Currently 1400 items per minute | Please lower your concurrency; tracker exceeded C10K |
|
21:05
๐
|
yipdw |
next person to use C10K in a sentence gets stabbed in the face over the internet |
|
21:06
๐
|
yipdw |
also wtf libavutil, where is av_frame_alloc |
|
21:07
๐
|
yipdw |
aaaaaaaaa: there's an issue sorta related to that -> https://github.com/ArchiveTeam/universal-tracker/issues/22 |
|
21:23
๐
|
aaaaaaaaa |
Hmm, looked through the bug tracker before I made mine, must have gone right over that one. |
|
21:24
๐
|
aaaaaaaaa |
In any case, my apologies for offending your sensibilities; I just copied the text from the current verizon tracker for my suggestion. |
|
21:31
๐
|
ersi |
C10K? |
|
21:47
๐
|
GitHub146 |
13seesaw-kit/06development 1468cd3d6 15Christopher Foo: warrior ui: Change broadcast msg indicator to darkgoldenrod... |
|
21:47
๐
|
GitHub146 |
13seesaw-kit/06development 147e51ee8 15Christopher Foo: warrior ui: Decrease header padding. Add .projectBroadcastMessage to styles.... |
|
21:47
๐
|
GitHub146 |
[13seesaw-kit] 15chfoo pushed 2 new commits to 06development: 02 https://github.com/ArchiveTeam/seesaw-kit/compare/a5ff73dbe94e...7e51ee832123 |
|
21:56
๐
|
GitHub47 |
13seesaw-kit/06development 1448f54e9 15Christopher Foo: web.py: Use MD5 digest as hash for indicator message... |
|
21:56
๐
|
GitHub47 |
[13seesaw-kit] 15chfoo pushed 1 new commit to 06development: 02 https://github.com/ArchiveTeam/seesaw-kit/commit/48f54e91b43cee3d237615e07ad05f29803624a8 |
|
22:17
๐
|
GitHub26 |
13seesaw-kit/06development 14043681a 15Anthony Eden: Add time indicator to each item. Shows how long it's been running. |
|
22:17
๐
|
GitHub26 |
13seesaw-kit/06development 14c25b066 15Christopher Foo: Merge branch 'temp1' into development... |
|
22:17
๐
|
GitHub26 |
[13seesaw-kit] 15chfoo pushed 2 new commits to 06development: 02 https://github.com/ArchiveTeam/seesaw-kit/compare/48f54e91b43c...c25b0669dd16 |
|
23:33
๐
|
GitHub124 |
13seesaw-kit/06development 1430456fb 15Christopher Foo: fixup! Merge branch 'temp1' into development... |
|
23:33
๐
|
GitHub124 |
13seesaw-kit/06development 146944392 15Christopher Foo: warrior ui: Remove 'Task Status Summary' h3 into a title attr. Fix align. |
|
23:33
๐
|
GitHub124 |
13seesaw-kit/06development 14c5d0911 15Christopher Foo: warrior ui: Move project name and timer under log. 'Runtime'รขยย'Elapsed'.... |
|
23:33
๐
|
GitHub124 |
[13seesaw-kit] 15chfoo pushed 3 new commits to 06development: 02 https://github.com/ArchiveTeam/seesaw-kit/compare/c25b0669dd16...c5d0911c27db |