[14:46] <SketchCow> So, it's time to nail some of this problem down. alard, if you're around.
[14:58] <undersco2> SketchCow: What's the problem?
[15:23] <SketchCow> Rsync issues.
[15:24] <SketchCow> I'm seeing a lot fail, would like to determine what it might be.
[15:33] <alard> SketchCow: Hi.
[15:45] <undersco2> SketchCow: What do you mean by rsync issues?
[15:47] <undersco2> or, rather, what's their failure mode?
[15:49] <alard> There's at least one problem I can think of: a combination of the rsync speed-limit option (closes the connection after 15 minutes at less than 1B/s) and the upload redirector (you're assigned a random upload server, which doesn't work too well with partial uploads).
[16:31] <SketchCow> Yes
[16:31] <SketchCow> The failing is happening constantly.
[16:31] <SketchCow> Not sure what to do here.
[16:31] <SketchCow> I can do more uploading
[16:31] <SketchCow> more slots
[16:33] <SketchCow> May 20 16:33:05 teamarchive0.fnf.archive.org rsyncd[10162]: rsync error: error in rsync protocol data stream (code 12) at io.c(605) [Receiver=3.0.9]
[16:33] <SketchCow> May 20 16:33:05 teamarchive0.fnf.archive.org rsyncd[10166]: rsync: connection unexpectedly closed (24 bytes received so far) [Receiver]
[16:33] <SketchCow> May 20 16:33:05 teamarchive0.fnf.archive.org rsyncd[10166]: rsync error: error in rsync protocol data stream (code 12) at io.c(605) [Receiver=3.0.9]
[16:33] <SketchCow> May 20 16:33:05 teamarchive0.fnf.archive.org rsyncd[10161]: rsync: connection unexpectedly closed (24 bytes received so far) [Receiver]
[16:33] <SketchCow> May 20 16:33:05 teamarchive0.fnf.archive.org rsyncd[10161]: rsync error: error in rsync protocol data stream (code 12) at io.c(605) [Receiver=3.0.9]
[16:33] <SketchCow> MA lot of those
[16:39] <undersco2> Hmm, I'm not getting any of those on my three upload servers
[16:39] <undersco2> Drive's not full, you don't have any weird firewall rules?
[17:04] <SketchCow> top - 17:03:04 up 3 days, 17:19,  2 users,  load average: 599.88, 577.53, 525.71
[17:04] <SketchCow> he heeeeee
[17:06] <Smiley> lol
[17:07] <SketchCow> Tasks: 1379 total,   1 running, 1378 sleeping,   0 stopped,   0 zombie
[17:09] <undersco2> It's weird your load is so high
[17:09] <undersco2> but I guess the server is doing a lot of other things at the same time?
[17:10] <undersco2> (weird becasue the bt-download box, which is handling >400 rsyncs atm, has a load of  ~6-10
[17:10] <undersco2> )
[17:10] <SketchCow> Let's check the mrtg
[17:11] <SketchCow> Won't push out
[17:11] <undersco2> Weird, it's doing barely anything network-wise
[17:11] <undersco2> Fair bit of disk IO though
[17:11] <Smiley> disk IO can cause massive load due to IOWait
[17:12] <Smiley> baiscally every cycle that your thread is waiting on another threads IO causes load+1
[17:12] <Smiley> So 200threads, all waiting for 1 threads IO = +200 load
[17:12] <undersco2> Maybe that's part of it, then
[17:12] <Smiley> took me a long time to understand that
[17:12] <Smiley> As most people think it's just CPU.
[17:15] <SketchCow> Here's the thing.
[17:15] <SketchCow> So, I can keep the links small
[17:15] <SketchCow> But we basically are at the behest of slow uploads
[17:19] <undersco2> SketchCow: At the moment, no new people are getting redirected to fos, fyi
[17:21] <SketchCow> Really.
[17:21] <SketchCow> You snagged all that?
[17:21] <SketchCow> Works for me, FOS is only so powerful
[17:26] <undersco2> mhm, I brought up 3 machines that were idle due to waiting on some other project stuff to go through and get the go-ahead
[17:27] <undersco2> and they seem to be handling it okay
[17:27] <undersco2> They aren't doing anything else like statusboard, et al. so that probably really helps