#warrior 2013-05-20,Mon

↑back Search

Time Nickname Message
14:46 🔗 SketchCow So, it's time to nail some of this problem down. alard, if you're around.
14:58 🔗 undersco2 SketchCow: What's the problem?
15:23 🔗 SketchCow Rsync issues.
15:24 🔗 SketchCow I'm seeing a lot fail, would like to determine what it might be.
15:33 🔗 alard SketchCow: Hi.
15:45 🔗 undersco2 SketchCow: What do you mean by rsync issues?
15:47 🔗 undersco2 or, rather, what's their failure mode?
15:49 🔗 alard There's at least one problem I can think of: a combination of the rsync speed-limit option (closes the connection after 15 minutes at less than 1B/s) and the upload redirector (you're assigned a random upload server, which doesn't work too well with partial uploads).
16:31 🔗 SketchCow Yes
16:31 🔗 SketchCow The failing is happening constantly.
16:31 🔗 SketchCow Not sure what to do here.
16:31 🔗 SketchCow I can do more uploading
16:31 🔗 SketchCow more slots
16:33 🔗 SketchCow May 20 16:33:05 teamarchive0.fnf.archive.org rsyncd[10162]: rsync error: error in rsync protocol data stream (code 12) at io.c(605) [Receiver=3.0.9]
16:33 🔗 SketchCow May 20 16:33:05 teamarchive0.fnf.archive.org rsyncd[10166]: rsync: connection unexpectedly closed (24 bytes received so far) [Receiver]
16:33 🔗 SketchCow May 20 16:33:05 teamarchive0.fnf.archive.org rsyncd[10166]: rsync error: error in rsync protocol data stream (code 12) at io.c(605) [Receiver=3.0.9]
16:33 🔗 SketchCow May 20 16:33:05 teamarchive0.fnf.archive.org rsyncd[10161]: rsync: connection unexpectedly closed (24 bytes received so far) [Receiver]
16:33 🔗 SketchCow May 20 16:33:05 teamarchive0.fnf.archive.org rsyncd[10161]: rsync error: error in rsync protocol data stream (code 12) at io.c(605) [Receiver=3.0.9]
16:33 🔗 SketchCow MA lot of those
16:39 🔗 undersco2 Hmm, I'm not getting any of those on my three upload servers
16:39 🔗 undersco2 Drive's not full, you don't have any weird firewall rules?
17:04 🔗 SketchCow top - 17:03:04 up 3 days, 17:19, 2 users, load average: 599.88, 577.53, 525.71
17:04 🔗 SketchCow he heeeeee
17:06 🔗 Smiley lol
17:07 🔗 SketchCow Tasks: 1379 total, 1 running, 1378 sleeping, 0 stopped, 0 zombie
17:09 🔗 undersco2 It's weird your load is so high
17:09 🔗 undersco2 but I guess the server is doing a lot of other things at the same time?
17:10 🔗 undersco2 (weird becasue the bt-download box, which is handling >400 rsyncs atm, has a load of ~6-10
17:10 🔗 undersco2 )
17:10 🔗 SketchCow Let's check the mrtg
17:11 🔗 SketchCow Won't push out
17:11 🔗 undersco2 Weird, it's doing barely anything network-wise
17:11 🔗 undersco2 Fair bit of disk IO though
17:11 🔗 Smiley disk IO can cause massive load due to IOWait
17:12 🔗 Smiley baiscally every cycle that your thread is waiting on another threads IO causes load+1
17:12 🔗 Smiley So 200threads, all waiting for 1 threads IO = +200 load
17:12 🔗 undersco2 Maybe that's part of it, then
17:12 🔗 Smiley took me a long time to understand that
17:12 🔗 Smiley As most people think it's just CPU.
17:15 🔗 SketchCow Here's the thing.
17:15 🔗 SketchCow So, I can keep the links small
17:15 🔗 SketchCow But we basically are at the behest of slow uploads
17:19 🔗 undersco2 SketchCow: At the moment, no new people are getting redirected to fos, fyi
17:21 🔗 SketchCow Really.
17:21 🔗 SketchCow You snagged all that?
17:21 🔗 SketchCow Works for me, FOS is only so powerful
17:26 🔗 undersco2 mhm, I brought up 3 machines that were idle due to waiting on some other project stuff to go through and get the go-ahead
17:27 🔗 undersco2 and they seem to be handling it okay
17:27 🔗 undersco2 They aren't doing anything else like statusboard, et al. so that probably really helps

irclogger-viewer