[02:46] <hneio_> ehhh
[02:46] <hneio_> I'll just bump it up to 4gb
[02:47] <hneio_> rather than run fewer threads
[15:26] <PepsiMax> ERROR: max connections (200) reached -- try again later
[15:26] <PepsiMax> Both machines where stuck at this error
[16:51] <ersi> PepsiMax: Where do you see that? Under RsyncUpload on a project?
[16:56] <PepsiMax> ersi: yep.
[16:56] <PepsiMax> It's trying to send it's collected data
[16:57] <ersi> Then don't worry. It'll obviously re-try to do that :)
[16:57] <PepsiMax> Well, it did like, a hundered times already.
[16:57] <PepsiMax> Every time it keeps failing
[16:58] <ersi> Well, there's nothing wrong with your warrior. The upload machine is maxed out on connections.
[16:59] <PepsiMax> Ah, so it wasn't a connection limit at my side.
[16:59] <ersi> No, I'm totally super certain about it. :)
[17:58] <PepsiMax> @ERROR: max connections (200) reached -- try again later
[17:58] <PepsiMax> rsync error: error starting client-server protocol (code 5) at main.c(1524) [sender=3.0.7]
[17:58] <PepsiMax> Process RsyncUpload returned exit code 5 for Item fanningbrinkman76.posterous.com
[17:58] <PepsiMax> ersi: well, how long is this going to last?
[17:58] <PepsiMax> All threads stuck now.
[17:58] <ersi> Calm down. I don't know.
[18:09] <PepsiMax> :(:(:(:(:(
[18:58] <ersi> PepsiMax: I'm able to upload occationally
[18:58] <ersi> So I guess it's easing up
[19:00] <PepsiMax> :-)
[19:19] <alard> Does SketchCow know about that upload limit?
[19:28] <ersi> Not sure. Likely, but I don't think so.
[20:59] <SketchCow> ?
[21:00] <SketchCow> Goodness, are we hitting that?
[21:00] <SmileyG> we broke fos maybe?
[21:00] <SketchCow> That machine is really slowing down.
[21:01] <SketchCow> [1289694.276295] TCP: Possible SYN flooding on port 873. Sending cookies.  Check SNMP counters.
[21:03] <SketchCow> http://teamarchive0.fnf.archive.org:8088/mrtg/
[21:07] <SketchCow> If I make that more than 200 connections, I feel like that won't help the machine.
[21:09] <SketchCow> I set it to 500.
[21:09] <SketchCow> We maxed it out in 60 seconds.
[21:10] <SketchCow> I'd call that "Something is wrong"
[21:12] <SketchCow> I set it to unlimited
[21:13] <SketchCow> Because I'm like that.
[21:13] <SketchCow> So, a metrix assload of errors are happening.
[21:16] <SketchCow> root@teamarchive0:/etc# ps -ax | grep rsync | wc -l
[21:16] <SketchCow> 1368
[21:16] <SketchCow> Just so you know.
[21:16] <SketchCow> 1,368 simultaneous Rsyncs.
[21:23] <SketchCow> 1771
[21:23] <SketchCow> root@teamarchive0:/etc# ps -ef | grep rsync | wc -l
[21:23] <SketchCow> Yeah, this is going to go crazy
[21:25] <ersi> Well, something is obviously misbehaving
[21:26] <ersi> Got any logs? Like where them there rsyncs are loadin' data?
[21:26] <ersi> SketchCow: the #archiveteam ping was for this ^
[21:26] <SketchCow> May 15 21:26:24 teamarchive0.fnf.archive.org rsyncd[10753]: rsync error: error in rsync protocol data stream (code 12) at io.c(605) [Receiver=3.0.9]
[21:26] <SketchCow> May 15 21:26:24 teamarchive0.fnf.archive.org rsyncd[10755]: rsync error: error in rsync protocol data stream (code 12) at io.c(605) [Receiver=3.0.9]
[21:26] <SketchCow> May 15 21:26:24 teamarchive0.fnf.archive.org rsyncd[10756]: rsync error: error in rsync protocol data stream (code 12) at io.c(605) [Receiver=3.0.9]
[21:38] <SketchCow> alard: Please remove underscor from the downloads for now.
[21:38] <SketchCow> I'm blocking all his connections.
[21:38] <SketchCow> ALL of them are failing, and it's murdering the system.
[21:47] <hneio_> I'm only getting 20kBps up
[21:47] <hneio_> is that your end or mine?
[21:47] <hneio_> Item EmilVester: Step 7 of 8 1146421248 65% 20.21kB/s 8:23:57
[21:47] <hneio_> urgh...
[21:48] <underscor> SketchCow: Hey, what's up?
[21:48] <underscor> What box is failing?
[21:48] <underscor> Have some IPs?
[21:48] <ersi> hneio_: It's not on your end.
[21:49] <ersi> hneio_: The upload machine is feeling a little barfy.
[21:50] <hneio_> lol...
[21:51] <hneio_> I have 8 hours left on that one, and 3 other finished Rsyncuploads
[21:51] <hneio_> so I'm running 6 threads, but only 2 are downloading
[21:53] <SketchCow> underscor: All of them.
[21:53] <SketchCow> Stop them all and let's step back
[21:54] <underscor> How were they failing?
[21:54] <SketchCow> Sending non functioning rsync connections.
[21:54] <SketchCow> Something's wrong with it
[21:54] <underscor> Can you whitelist one of the things so I can see what my end is saying?
[21:55] <underscor> If I stop them, I'm going to lose ~400GB of inflight data
[21:55] <underscor> (which I can do, but is not optimal)
[21:56] <underscor> (I'm watching bt-download03.us.archive.org / 207.241.226.32 if you want to un-blacklist that one)
[21:56] <underscor> and I'll see what console is saying
[21:57] <ersi> hneio_: My uploads are feeling fine now and are over 300KB/s now
[21:57] <hneio_> yes, back up
[21:59] <SketchCow> May 15 21:59:06 teamarchive0.fnf.archive.org rsyncd[2238]: rsync to alardland/warrior/posterous/underscor/ from bt-download08.us.archive.org (207.241.239.196)
[21:59] <SketchCow> May 15 21:59:06 teamarchive0.fnf.archive.org rsyncd[2238]: receiving file list
[21:59] <SketchCow> May 15 21:59:06 teamarchive0.fnf.archive.org rsyncd[2238]: rsync: connection unexpectedly closed (24 bytes received so far) [Receiver]
[21:59] <SketchCow> May 15 21:59:06 teamarchive0.fnf.archive.org rsyncd[2238]: rsync error: error in rsync protocol data stream (code 12) at io.c(605) [Receiver=3.0.9]
[21:59] <SketchCow> May 15 21:59:07 teamarchive0.fnf.archive.org rsyncd[2240]: connect from cpc23-cove12-2-0-cust135.3-1.cable.virginmedia.com (86.20.136.136)
[21:59] <SketchCow> May 15 21:59:07 teamarchive0.fnf.archive.org rsyncd[2241]: connect from bt-download10.us.archive.org (207.241.239.194)
[21:59] <SketchCow> May 15 21:59:07 teamarchive0.fnf.archive.org rsyncd[2241]: rsync to alardland/warrior/posterous/underscor/ from bt-download10.us.archive.org (207.241.239.194)
[21:59] <SketchCow> May 15 21:59:07 teamarchive0.fnf.archive.org rsyncd[2241]: receiving file list
[21:59] <SketchCow> May 15 21:59:07 teamarchive0.fnf.archive.org rsyncd[2242]: connect from bt-download11.us.archive.org (207.241.239.193)
[21:59] <SketchCow> May 15 21:59:07 teamarchive0.fnf.archive.org rsyncd[2245]: connect from bt-download10.us.archive.org (207.241.239.194)
[21:59] <SketchCow> May 15 21:59:07 teamarchive0.fnf.archive.org rsyncd[2242]: rsync to alardland/warrior/posterous/underscor/ from bt-download11.us.archive.org (207.241.
[22:01] <underscor> SketchCow: 08 was turbofucked
[22:01] <underscor> Checking 09 now
[22:01] <underscor> 09 seems healthy, although banned again
[22:02] <SketchCow> Oh yeah, they're all banned until you scrub the skunk out of ALL these dogs
[22:02] <SketchCow> Let me know when that's done.
[22:26] <underscor> SketchCow: There are 4 machines with active inflight data that are attempting to upload
[22:26] <underscor> I am actively watching all four
[22:26] <underscor> They seem healthy
[22:26] <underscor> Can you unban while I watch them?
[22:28] <SketchCow> May 15 22:28:38 teamarchive0.fnf.archive.org rsyncd[20014]: rsync error: error in rsync protocol data stream (code 12) at io.c(605) [Receiver=3.0.9]
[22:28] <SketchCow> May 15 22:28:38 teamarchive0.fnf.archive.org rsyncd[20015]: rsync: connection unexpectedly closed (24 bytes received so far) [Receiver]
[22:28] <SketchCow> May 15 22:28:38 teamarchive0.fnf.archive.org rsyncd[20015]: rsync error: error in rsync protocol data stream (code 12) at io.c(605) [Receiver=3.0.9]
[22:28] <SketchCow> May 15 22:28:38 teamarchive0.fnf.archive.org rsyncd[20016]: rsync: connection unexpectedly closed (24 bytes received so far) [Receiver]
[22:28] <SketchCow> May 15 22:28:38 teamarchive0.fnf.archive.org rsyncd[20016]: rsync error: error in rsync protocol data stream (code 12) at io.c(605) [Receiver=3.0.9]
[22:29] <SketchCow> May 15 22:28:39 teamarchive0.fnf.archive.org rsyncd[20017]: connect from bt-download09.us.archive.org (207.241.239.195)
[22:29] <SketchCow> May 15 22:28:39 teamarchive0.fnf.archive.org rsyncd[20017]: rsync to alardland/warrior/posterous/underscor/ from bt-download09.us.archive.org (207.241.239.195)
[22:29] <SketchCow> Failing failing failing.
[22:29] <SketchCow> Up goes the ban.
[22:29] <SketchCow> Plus when it's not being denied, jesus
[22:29] <SketchCow> This machine is so hosed that typing vi /etc/rsyncd.conf makes it wait 12 seconds.
[22:29] <SketchCow> 12. seconds.
[22:30] <underscor> [sender=3.0.9]
[22:30] <underscor> or directory (2)
[22:30] <underscor> rsync error: errors selecting input/output files, dirs (code 3) at flist.c(2027)
[22:30] <underscor> rsync: change_dir "/1/posterous-grab/data/46239248-50358/" failed: No such file
[22:30] <underscor> sending incremental file list
[22:30] <underscor> Process RsyncUpload returned exit code 3 for Item pcenoppislent.posterous.com
[22:31] <SketchCow> Why is it trying to send it there.
[22:31] <SketchCow> How old is that client?
[22:31] <underscor> Looks like the datadir got fucked, I guess
[22:31] <underscor> No, no
[22:31] <underscor> that's local
[22:32] <underscor> I'm just going to cut my losses and release my claims, because the data directory in my posterous-grab folder is inconsistent
[22:41] <underscor> SketchCow: Everything stopped
[22:42] <underscor> should be*
[22:42] <underscor> Unban?
[22:45] <SketchCow> I should make you wait a day.
[22:45] <SketchCow> But fine.
[23:36] <hneio_> upload speed dropped off again
[23:38] <SketchCow> Yeah, well.
[23:38] <SketchCow> News flash: it's going to be a back and forth day
[23:38] <hneio_> I'm seeding a tails iso at 20kBps
[23:38] <hneio_> and uploading to archiveteam at 5kBps
[23:39] <SketchCow> Yes.
[23:39] <SketchCow> The machine has 3,500 connections to it right now.
[23:39] <hneio_> ouch
[23:40] <hneio_> well, I suppose that's a good thing