| Time |
Nickname |
Message |
|
02:46
🔗
|
hneio_ |
ehhh |
|
02:46
🔗
|
hneio_ |
I'll just bump it up to 4gb |
|
02:47
🔗
|
hneio_ |
rather than run fewer threads |
|
15:26
🔗
|
PepsiMax |
ERROR: max connections (200) reached -- try again later |
|
15:26
🔗
|
PepsiMax |
Both machines where stuck at this error |
|
16:51
🔗
|
ersi |
PepsiMax: Where do you see that? Under RsyncUpload on a project? |
|
16:56
🔗
|
PepsiMax |
ersi: yep. |
|
16:56
🔗
|
PepsiMax |
It's trying to send it's collected data |
|
16:57
🔗
|
ersi |
Then don't worry. It'll obviously re-try to do that :) |
|
16:57
🔗
|
PepsiMax |
Well, it did like, a hundered times already. |
|
16:57
🔗
|
PepsiMax |
Every time it keeps failing |
|
16:58
🔗
|
ersi |
Well, there's nothing wrong with your warrior. The upload machine is maxed out on connections. |
|
16:59
🔗
|
PepsiMax |
Ah, so it wasn't a connection limit at my side. |
|
16:59
🔗
|
ersi |
No, I'm totally super certain about it. :) |
|
17:58
🔗
|
PepsiMax |
@ERROR: max connections (200) reached -- try again later |
|
17:58
🔗
|
PepsiMax |
rsync error: error starting client-server protocol (code 5) at main.c(1524) [sender=3.0.7] |
|
17:58
🔗
|
PepsiMax |
Process RsyncUpload returned exit code 5 for Item fanningbrinkman76.posterous.com |
|
17:58
🔗
|
PepsiMax |
ersi: well, how long is this going to last? |
|
17:58
🔗
|
PepsiMax |
All threads stuck now. |
|
17:58
🔗
|
ersi |
Calm down. I don't know. |
|
18:09
🔗
|
PepsiMax |
:(:(:(:(:( |
|
18:58
🔗
|
ersi |
PepsiMax: I'm able to upload occationally |
|
18:58
🔗
|
ersi |
So I guess it's easing up |
|
19:00
🔗
|
PepsiMax |
:-) |
|
19:19
🔗
|
alard |
Does SketchCow know about that upload limit? |
|
19:28
🔗
|
ersi |
Not sure. Likely, but I don't think so. |
|
20:59
🔗
|
SketchCow |
? |
|
21:00
🔗
|
SketchCow |
Goodness, are we hitting that? |
|
21:00
🔗
|
SmileyG |
we broke fos maybe? |
|
21:00
🔗
|
SketchCow |
That machine is really slowing down. |
|
21:01
🔗
|
SketchCow |
[1289694.276295] TCP: Possible SYN flooding on port 873. Sending cookies. Check SNMP counters. |
|
21:03
🔗
|
SketchCow |
http://teamarchive0.fnf.archive.org:8088/mrtg/ |
|
21:07
🔗
|
SketchCow |
If I make that more than 200 connections, I feel like that won't help the machine. |
|
21:09
🔗
|
SketchCow |
I set it to 500. |
|
21:09
🔗
|
SketchCow |
We maxed it out in 60 seconds. |
|
21:10
🔗
|
SketchCow |
I'd call that "Something is wrong" |
|
21:12
🔗
|
SketchCow |
I set it to unlimited |
|
21:13
🔗
|
SketchCow |
Because I'm like that. |
|
21:13
🔗
|
SketchCow |
So, a metrix assload of errors are happening. |
|
21:16
🔗
|
SketchCow |
root@teamarchive0:/etc# ps -ax | grep rsync | wc -l |
|
21:16
🔗
|
SketchCow |
1368 |
|
21:16
🔗
|
SketchCow |
Just so you know. |
|
21:16
🔗
|
SketchCow |
1,368 simultaneous Rsyncs. |
|
21:23
🔗
|
SketchCow |
1771 |
|
21:23
🔗
|
SketchCow |
root@teamarchive0:/etc# ps -ef | grep rsync | wc -l |
|
21:23
🔗
|
SketchCow |
Yeah, this is going to go crazy |
|
21:25
🔗
|
ersi |
Well, something is obviously misbehaving |
|
21:26
🔗
|
ersi |
Got any logs? Like where them there rsyncs are loadin' data? |
|
21:26
🔗
|
ersi |
SketchCow: the #archiveteam ping was for this ^ |
|
21:26
🔗
|
SketchCow |
May 15 21:26:24 teamarchive0.fnf.archive.org rsyncd[10753]: rsync error: error in rsync protocol data stream (code 12) at io.c(605) [Receiver=3.0.9] |
|
21:26
🔗
|
SketchCow |
May 15 21:26:24 teamarchive0.fnf.archive.org rsyncd[10755]: rsync error: error in rsync protocol data stream (code 12) at io.c(605) [Receiver=3.0.9] |
|
21:26
🔗
|
SketchCow |
May 15 21:26:24 teamarchive0.fnf.archive.org rsyncd[10756]: rsync error: error in rsync protocol data stream (code 12) at io.c(605) [Receiver=3.0.9] |
|
21:38
🔗
|
SketchCow |
alard: Please remove underscor from the downloads for now. |
|
21:38
🔗
|
SketchCow |
I'm blocking all his connections. |
|
21:38
🔗
|
SketchCow |
ALL of them are failing, and it's murdering the system. |
|
21:47
🔗
|
hneio_ |
I'm only getting 20kBps up |
|
21:47
🔗
|
hneio_ |
is that your end or mine? |
|
21:47
🔗
|
hneio_ |
Item EmilVester: Step 7 of 8 1146421248 65% 20.21kB/s 8:23:57 |
|
21:47
🔗
|
hneio_ |
urgh... |
|
21:48
🔗
|
underscor |
SketchCow: Hey, what's up? |
|
21:48
🔗
|
underscor |
What box is failing? |
|
21:48
🔗
|
underscor |
Have some IPs? |
|
21:48
🔗
|
ersi |
hneio_: It's not on your end. |
|
21:49
🔗
|
ersi |
hneio_: The upload machine is feeling a little barfy. |
|
21:50
🔗
|
hneio_ |
lol... |
|
21:51
🔗
|
hneio_ |
I have 8 hours left on that one, and 3 other finished Rsyncuploads |
|
21:51
🔗
|
hneio_ |
so I'm running 6 threads, but only 2 are downloading |
|
21:53
🔗
|
SketchCow |
underscor: All of them. |
|
21:53
🔗
|
SketchCow |
Stop them all and let's step back |
|
21:54
🔗
|
underscor |
How were they failing? |
|
21:54
🔗
|
SketchCow |
Sending non functioning rsync connections. |
|
21:54
🔗
|
SketchCow |
Something's wrong with it |
|
21:54
🔗
|
underscor |
Can you whitelist one of the things so I can see what my end is saying? |
|
21:55
🔗
|
underscor |
If I stop them, I'm going to lose ~400GB of inflight data |
|
21:55
🔗
|
underscor |
(which I can do, but is not optimal) |
|
21:56
🔗
|
underscor |
(I'm watching bt-download03.us.archive.org / 207.241.226.32 if you want to un-blacklist that one) |
|
21:56
🔗
|
underscor |
and I'll see what console is saying |
|
21:57
🔗
|
ersi |
hneio_: My uploads are feeling fine now and are over 300KB/s now |
|
21:57
🔗
|
hneio_ |
yes, back up |
|
21:59
🔗
|
SketchCow |
May 15 21:59:06 teamarchive0.fnf.archive.org rsyncd[2238]: rsync to alardland/warrior/posterous/underscor/ from bt-download08.us.archive.org (207.241.239.196) |
|
21:59
🔗
|
SketchCow |
May 15 21:59:06 teamarchive0.fnf.archive.org rsyncd[2238]: receiving file list |
|
21:59
🔗
|
SketchCow |
May 15 21:59:06 teamarchive0.fnf.archive.org rsyncd[2238]: rsync: connection unexpectedly closed (24 bytes received so far) [Receiver] |
|
21:59
🔗
|
SketchCow |
May 15 21:59:06 teamarchive0.fnf.archive.org rsyncd[2238]: rsync error: error in rsync protocol data stream (code 12) at io.c(605) [Receiver=3.0.9] |
|
21:59
🔗
|
SketchCow |
May 15 21:59:07 teamarchive0.fnf.archive.org rsyncd[2240]: connect from cpc23-cove12-2-0-cust135.3-1.cable.virginmedia.com (86.20.136.136) |
|
21:59
🔗
|
SketchCow |
May 15 21:59:07 teamarchive0.fnf.archive.org rsyncd[2241]: connect from bt-download10.us.archive.org (207.241.239.194) |
|
21:59
🔗
|
SketchCow |
May 15 21:59:07 teamarchive0.fnf.archive.org rsyncd[2241]: rsync to alardland/warrior/posterous/underscor/ from bt-download10.us.archive.org (207.241.239.194) |
|
21:59
🔗
|
SketchCow |
May 15 21:59:07 teamarchive0.fnf.archive.org rsyncd[2241]: receiving file list |
|
21:59
🔗
|
SketchCow |
May 15 21:59:07 teamarchive0.fnf.archive.org rsyncd[2242]: connect from bt-download11.us.archive.org (207.241.239.193) |
|
21:59
🔗
|
SketchCow |
May 15 21:59:07 teamarchive0.fnf.archive.org rsyncd[2245]: connect from bt-download10.us.archive.org (207.241.239.194) |
|
21:59
🔗
|
SketchCow |
May 15 21:59:07 teamarchive0.fnf.archive.org rsyncd[2242]: rsync to alardland/warrior/posterous/underscor/ from bt-download11.us.archive.org (207.241. |
|
22:01
🔗
|
underscor |
SketchCow: 08 was turbofucked |
|
22:01
🔗
|
underscor |
Checking 09 now |
|
22:01
🔗
|
underscor |
09 seems healthy, although banned again |
|
22:02
🔗
|
SketchCow |
Oh yeah, they're all banned until you scrub the skunk out of ALL these dogs |
|
22:02
🔗
|
SketchCow |
Let me know when that's done. |
|
22:26
🔗
|
underscor |
SketchCow: There are 4 machines with active inflight data that are attempting to upload |
|
22:26
🔗
|
underscor |
I am actively watching all four |
|
22:26
🔗
|
underscor |
They seem healthy |
|
22:26
🔗
|
underscor |
Can you unban while I watch them? |
|
22:28
🔗
|
SketchCow |
May 15 22:28:38 teamarchive0.fnf.archive.org rsyncd[20014]: rsync error: error in rsync protocol data stream (code 12) at io.c(605) [Receiver=3.0.9] |
|
22:28
🔗
|
SketchCow |
May 15 22:28:38 teamarchive0.fnf.archive.org rsyncd[20015]: rsync: connection unexpectedly closed (24 bytes received so far) [Receiver] |
|
22:28
🔗
|
SketchCow |
May 15 22:28:38 teamarchive0.fnf.archive.org rsyncd[20015]: rsync error: error in rsync protocol data stream (code 12) at io.c(605) [Receiver=3.0.9] |
|
22:28
🔗
|
SketchCow |
May 15 22:28:38 teamarchive0.fnf.archive.org rsyncd[20016]: rsync: connection unexpectedly closed (24 bytes received so far) [Receiver] |
|
22:28
🔗
|
SketchCow |
May 15 22:28:38 teamarchive0.fnf.archive.org rsyncd[20016]: rsync error: error in rsync protocol data stream (code 12) at io.c(605) [Receiver=3.0.9] |
|
22:29
🔗
|
SketchCow |
May 15 22:28:39 teamarchive0.fnf.archive.org rsyncd[20017]: connect from bt-download09.us.archive.org (207.241.239.195) |
|
22:29
🔗
|
SketchCow |
May 15 22:28:39 teamarchive0.fnf.archive.org rsyncd[20017]: rsync to alardland/warrior/posterous/underscor/ from bt-download09.us.archive.org (207.241.239.195) |
|
22:29
🔗
|
SketchCow |
Failing failing failing. |
|
22:29
🔗
|
SketchCow |
Up goes the ban. |
|
22:29
🔗
|
SketchCow |
Plus when it's not being denied, jesus |
|
22:29
🔗
|
SketchCow |
This machine is so hosed that typing vi /etc/rsyncd.conf makes it wait 12 seconds. |
|
22:29
🔗
|
SketchCow |
12. seconds. |
|
22:30
🔗
|
underscor |
[sender=3.0.9] |
|
22:30
🔗
|
underscor |
or directory (2) |
|
22:30
🔗
|
underscor |
rsync error: errors selecting input/output files, dirs (code 3) at flist.c(2027) |
|
22:30
🔗
|
underscor |
rsync: change_dir "/1/posterous-grab/data/46239248-50358/" failed: No such file |
|
22:30
🔗
|
underscor |
sending incremental file list |
|
22:30
🔗
|
underscor |
Process RsyncUpload returned exit code 3 for Item pcenoppislent.posterous.com |
|
22:31
🔗
|
SketchCow |
Why is it trying to send it there. |
|
22:31
🔗
|
SketchCow |
How old is that client? |
|
22:31
🔗
|
underscor |
Looks like the datadir got fucked, I guess |
|
22:31
🔗
|
underscor |
No, no |
|
22:31
🔗
|
underscor |
that's local |
|
22:32
🔗
|
underscor |
I'm just going to cut my losses and release my claims, because the data directory in my posterous-grab folder is inconsistent |
|
22:41
🔗
|
underscor |
SketchCow: Everything stopped |
|
22:42
🔗
|
underscor |
should be* |
|
22:42
🔗
|
underscor |
Unban? |
|
22:45
🔗
|
SketchCow |
I should make you wait a day. |
|
22:45
🔗
|
SketchCow |
But fine. |
|
23:36
🔗
|
hneio_ |
upload speed dropped off again |
|
23:38
🔗
|
SketchCow |
Yeah, well. |
|
23:38
🔗
|
SketchCow |
News flash: it's going to be a back and forth day |
|
23:38
🔗
|
hneio_ |
I'm seeding a tails iso at 20kBps |
|
23:38
🔗
|
hneio_ |
and uploading to archiveteam at 5kBps |
|
23:39
🔗
|
SketchCow |
Yes. |
|
23:39
🔗
|
SketchCow |
The machine has 3,500 connections to it right now. |
|
23:39
🔗
|
hneio_ |
ouch |
|
23:40
🔗
|
hneio_ |
well, I suppose that's a good thing |