[00:14] All right, calling it, going to keep it at 1,000 [00:19] http://teamarchive0.fnf.archive.org:8088/mrtg/committed-day.png tells the story. [00:31] geez [00:49] SketchCow: thanks for unban [00:53] also, wow, we're keeping fos busy [00:57] I'm no longer even uploading [00:57] not one of the thousand [00:57] luckily I still have 2 threads left which haven't finished downloading [01:00] We're truly hosing this machine. [01:00] I may kill rsync and try again [01:04] hello everyone [01:04] my warrior's having some issues [01:04] upload problems? [01:04] rsync, yeah [01:04] the server's under heavy load [01:04] just wait [01:04] I figured [01:04] http://teamarchive0.fnf.archive.org:8088/mrtg/committed-day.png [01:04] just checking in [01:04] look at that [01:04] damn [01:05] that's all legitimate warrior traffic? [01:05] or is something else going on [01:06] SketchCow ^ [01:07] Hneio, you linked rabid_mar to the memory usage of the server.. [01:08] yorsh [01:08] that was the idea [01:09] I thought it was disk io [01:09] don't mind me [01:10] that's here http://teamarchive0.fnf.archive.org:8088/mrtg/diskv3-day.png [01:13] I found the page with everything on it [01:13] out of curiosity, is this just elevated usage or is there something else going on with the server? [01:14] or are you guys out of the loop because this is one of archive.org's machines [01:15] Yes [01:16] this was a case where we had one of the members who was relatively energetic have machines pounding the machine [01:16] man... 5 finished RSyncUploads [01:17] only one thread left downloading [01:18] gotcha, thanks for the info [01:19] 5457 [01:19] 4709 [01:19] 2372 [01:20] So good. [01:25] now I'm sending incremental file list [01:25] we'll see if that goes through... [13:56] Is fos still on fire? [14:01] Yes. Also, the issue was discussed in #warrior and #preposterus earlier. Maybe keep it somewhere there. [14:01] * Smiley poits to the channel name [14:02] Oh snap, you got me [14:02] Only other place that seems sensible is the tracker admin channel, but that's kind private :P [14:02] I did make you look, though? :D [14:02] haha yes I did :D [14:02] Was wondering if my irssi wasn't redrawing correctly [14:02] mwahaha! Let's say.. that was my plan all along! [15:28] http://i.imgur.com/ERtArKf.png [15:29] The #warrior channel is the place to discuss the warrior. [15:29] I've got it going now, it hit 400 connections. [15:29] Little I can do about this, I am NOT raising it right now. [15:29] That killed the machine. [15:29] top - 15:29:32 up 15 days, 20:10, 6 users, load average: 409.33, 391.02, 263.50 [15:29] See, it's still a little pissed now. [15:33] tbh loower it if it's an issue [15:33] the warriors can wait. [15:34] better they wait a bit than the box exploding. [19:39] So, seriously. [19:39] Right now we have underscor running well over a hundred processes on archive.org machines, jamming all their shit into one poor overworked public facing archive.org machine, to upload to archive.org. [19:39] I feel like.... I feel like maybe we're adding a useless step there. [19:40] I would like underscor and alard to talk and maybe have one of his ravenous insanity boxes play the part of relay to archive.org's machines. [19:42] SketchCow: kau [19:42] kay* [19:42] Hello. [19:42] Hey alard [19:42] Two of you. Work together. [19:43] It'll take you 9 seconds, you're geniuses [19:43] Because FOS does quintuple duty, if we can use an alternate machine, it works just as well. [19:43] If it was a dedicated relay, it'd probably be happy, but it runs statusboard, I have uploads going on, I have torrents going on, and all sorts of other things. [19:44] It can work fine as a general relay, but underscor specifically is just a tsunami of data. [19:44] And more importantly, connections. [19:44] He's doing 100-150 connections a second. [19:44] I mean minute. [19:45] alard: are the package-and-upload scripts easily portable? [19:45] https://github.com/ArchiveTeam/archiveteam-megawarc-factory [19:45] I have another box(es) I can run them on to split the load with fos [19:45] It should be relatively easy, I think. [19:45] Yes, use that. [19:45] Set up rsync, fill in the configuration file, go. [19:45] Do it with one of your boxes, make sure it's doing the right thing, then add more to it. [19:45] Don't just flop over, that's what a 19 year old does [19:46] Then you're all oil and flying parts [19:46] But I'm watching FOS snowball trying to keep up AND do the gzip/upload work [19:46] load average: 112.89, 115.99, 116.29 [19:46] Did you set any ionice priorities? [19:47] Not even one. [19:47] I could do that, it starts to get silly, though. [19:47] You could try to give the gzip/megawarc bits a lower priority. They'll grab everything they can. [19:47] It helped on short's server. [19:48] The core issue is we have 32 internal servers slamming another internal server when they could be 31 internal servers square-dancing with a specific server. [19:48] ionicing will work but the snowballing will still happen. [19:48] The drive this is on is 41% full, 9tb [19:48] It's all these two projects. [19:49] Nothing helps against 32 servers, no. [19:49] And when I go to watch the logs, it's just underscor slamming the shit out of the machine. [19:49] Are we doing 25 or 50gb chunks? [19:49] 50gb. [19:49] You've successfully made uploading robust enough for it to be on several servers, alard. We should use that. [19:49] I set it to 25GB for a while because some of the 50GB files weren't uploading well, but maybe that was just because of the disk error. [19:49] What's the collection ID? [19:50] and the item title pattern, item identifier prefix, and file prefix? [19:50] project? [19:50] posterous [19:50] Take it to privatew [19:51] Just slam through it, it'll take 10 minutes. [19:51] It's a very good script. [20:01] I am assuming it's running. [20:01] Guess what my hint is. [20:04] FOS doing better? [20:05] I'm bringing in data at 585mbps [20:05] Damn [20:06] Well, I just blocked you [20:06] Feels good, man [20:06] :c [20:07] Having done that, I'm increasing connections to 200 [20:07] Total DISK WRITE: 90.77M/s [20:07] Feels good, man [20:08] Yeah, you were DDOSing the living shit fuck out of FOS [20:08] * SketchCow renames it OS [20:08] DOS [20:08] Denial of Solitude [20:08] hahaha [20:08] I was DOSing the FOS? [20:08] HERE IN MY FORTRESS OF what the fuck are you doing [20:08] :D :D :D [20:08] Now I'm DOSing my own box hahaha [20:11] It has certainly made the tracker impressive [20:19] Nice, the machine is happy now I guess [20:19] Since the uploads are coming trough :-) [20:26] half the world just saw random uploads all start [20:26] they think skynet just booted off FOS. [20:26] It's consuming all the worlds infomation! [22:26] Oh god it's slow again :-( [22:27] :O [22:29] * Smiley pokes SketchCow [22:30] If it's just the same issue, it'll sort it's self out [22:32] nah [22:33] ersi tried to calm me down yesterday too [22:36] PepsiMax: huh? [22:37] hehe [22:37] 20:09:34 < PepsiMax> :(:(:(:(:( [22:37] 20:58:52 <@ersi> PepsiMax: I'm able to upload occationally [22:37] 20:58:56 <@ersi> So I guess it's easing up [22:48] PepsiMax: nod. [22:55] WHAAAAT [22:55] We're at max connections. 200. [22:56] Also, it's not slow, it's fast as fuck [22:56] It's that it's got too many people who want the teat [22:56] Now, if you want [22:56] I can fuck the machine up REALLY good, turn it ALL the way to 11 [22:56] Then you can feel like it's "faster" [22:56] while it goes into full melted slag [22:57] At this point, underscor does not use the machine. [22:57] So this is everyone else. [22:57] Some people are uploading slow. [22:57] If I add more, more goes slow. [22:57] Load average is at 200 [22:57] That's lagwagon territory [22:57] If I put it to 300, it'll fill at 300. [22:57] Etc. [22:57] Etc. [22:57] Etc. [22:57] Etc. [22:57] Alard worked very hard for a machine to automatically just "work" [22:58] It doesn't need you, the concerned sweaty meatbag, staring at it, hoping the magic dust will work. [22:58] I'm doing some other stuff to it internally, it might help. [22:58] meatbag lol [22:59] lol [22:59] Is that me or PepsiMax ? [22:59] XD [22:59] Both [22:59] all [22:59] Everyone [22:59] i am one of the slow uploaders^^ [22:59] people who don't even know about the project [22:59] lol [22:59] girls in bikinis. dogs in restaurants. [22:59] trees. [23:00] silverware piles. mushrooms. [23:00] Everyone gets a Paddlin' [23:00] We're shoving 400 pounds of shit into a 5 pound bag. I'm working with it. [23:00] But this is why we have error correction. [23:01] The big thing is people with 56k Wondermodems opening 6 instances to 'help' [23:01] Not much we can do about this, or eval it to stop that situation. [23:02] * Smiley glares at flaushy XD [23:02] hilarious. [23:02] a bucket of hilarious. [23:02] back to movie. [23:02] as long as that's all it is, and not something else melting [23:02] Every few minutes, the machine runs a java instance to generate a new book [23:02] for statusboard.archive.org. [23:02] That eats the CPU for a minute or two [23:02] Then back [23:02] Ahhhhhh ok [23:02] Just one of many things on there. [23:07] I keep getting a max connection message :/ [23:08] it will work eventually [23:08] just be patient [23:09] True, just frustrating at times. [23:50] underscor: Stop sending things to FOS [23:51] You are murdering that machine. [23:51] Even with a goddamn DENY, it's flooding the machine. [23:51] I reboot the whole damn thing, it's JUST running rsync [23:51] And when I turned it on, it dropped to basically nothing. [23:51] You're killing it. [23:51] Stop. [23:52] Jesus, what are you doing. [23:52] This thing is at 245 load. [23:52] From rsync hits. [23:52] Just rsync hits.