| Time |
Nickname |
Message |
|
14:46
🔗
|
SketchCow |
So, it's time to nail some of this problem down. alard, if you're around. |
|
14:58
🔗
|
undersco2 |
SketchCow: What's the problem? |
|
15:23
🔗
|
SketchCow |
Rsync issues. |
|
15:24
🔗
|
SketchCow |
I'm seeing a lot fail, would like to determine what it might be. |
|
15:33
🔗
|
alard |
SketchCow: Hi. |
|
15:45
🔗
|
undersco2 |
SketchCow: What do you mean by rsync issues? |
|
15:47
🔗
|
undersco2 |
or, rather, what's their failure mode? |
|
15:49
🔗
|
alard |
There's at least one problem I can think of: a combination of the rsync speed-limit option (closes the connection after 15 minutes at less than 1B/s) and the upload redirector (you're assigned a random upload server, which doesn't work too well with partial uploads). |
|
16:31
🔗
|
SketchCow |
Yes |
|
16:31
🔗
|
SketchCow |
The failing is happening constantly. |
|
16:31
🔗
|
SketchCow |
Not sure what to do here. |
|
16:31
🔗
|
SketchCow |
I can do more uploading |
|
16:31
🔗
|
SketchCow |
more slots |
|
16:33
🔗
|
SketchCow |
May 20 16:33:05 teamarchive0.fnf.archive.org rsyncd[10162]: rsync error: error in rsync protocol data stream (code 12) at io.c(605) [Receiver=3.0.9] |
|
16:33
🔗
|
SketchCow |
May 20 16:33:05 teamarchive0.fnf.archive.org rsyncd[10166]: rsync: connection unexpectedly closed (24 bytes received so far) [Receiver] |
|
16:33
🔗
|
SketchCow |
May 20 16:33:05 teamarchive0.fnf.archive.org rsyncd[10166]: rsync error: error in rsync protocol data stream (code 12) at io.c(605) [Receiver=3.0.9] |
|
16:33
🔗
|
SketchCow |
May 20 16:33:05 teamarchive0.fnf.archive.org rsyncd[10161]: rsync: connection unexpectedly closed (24 bytes received so far) [Receiver] |
|
16:33
🔗
|
SketchCow |
May 20 16:33:05 teamarchive0.fnf.archive.org rsyncd[10161]: rsync error: error in rsync protocol data stream (code 12) at io.c(605) [Receiver=3.0.9] |
|
16:33
🔗
|
SketchCow |
MA lot of those |
|
16:39
🔗
|
undersco2 |
Hmm, I'm not getting any of those on my three upload servers |
|
16:39
🔗
|
undersco2 |
Drive's not full, you don't have any weird firewall rules? |
|
17:04
🔗
|
SketchCow |
top - 17:03:04 up 3 days, 17:19, 2 users, load average: 599.88, 577.53, 525.71 |
|
17:04
🔗
|
SketchCow |
he heeeeee |
|
17:06
🔗
|
Smiley |
lol |
|
17:07
🔗
|
SketchCow |
Tasks: 1379 total, 1 running, 1378 sleeping, 0 stopped, 0 zombie |
|
17:09
🔗
|
undersco2 |
It's weird your load is so high |
|
17:09
🔗
|
undersco2 |
but I guess the server is doing a lot of other things at the same time? |
|
17:10
🔗
|
undersco2 |
(weird becasue the bt-download box, which is handling >400 rsyncs atm, has a load of ~6-10 |
|
17:10
🔗
|
undersco2 |
) |
|
17:10
🔗
|
SketchCow |
Let's check the mrtg |
|
17:11
🔗
|
SketchCow |
Won't push out |
|
17:11
🔗
|
undersco2 |
Weird, it's doing barely anything network-wise |
|
17:11
🔗
|
undersco2 |
Fair bit of disk IO though |
|
17:11
🔗
|
Smiley |
disk IO can cause massive load due to IOWait |
|
17:12
🔗
|
Smiley |
baiscally every cycle that your thread is waiting on another threads IO causes load+1 |
|
17:12
🔗
|
Smiley |
So 200threads, all waiting for 1 threads IO = +200 load |
|
17:12
🔗
|
undersco2 |
Maybe that's part of it, then |
|
17:12
🔗
|
Smiley |
took me a long time to understand that |
|
17:12
🔗
|
Smiley |
As most people think it's just CPU. |
|
17:15
🔗
|
SketchCow |
Here's the thing. |
|
17:15
🔗
|
SketchCow |
So, I can keep the links small |
|
17:15
🔗
|
SketchCow |
But we basically are at the behest of slow uploads |
|
17:19
🔗
|
undersco2 |
SketchCow: At the moment, no new people are getting redirected to fos, fyi |
|
17:21
🔗
|
SketchCow |
Really. |
|
17:21
🔗
|
SketchCow |
You snagged all that? |
|
17:21
🔗
|
SketchCow |
Works for me, FOS is only so powerful |
|
17:26
🔗
|
undersco2 |
mhm, I brought up 3 machines that were idle due to waiting on some other project stuff to go through and get the go-ahead |
|
17:27
🔗
|
undersco2 |
and they seem to be handling it okay |
|
17:27
🔗
|
undersco2 |
They aren't doing anything else like statusboard, et al. so that probably really helps |