#warrior 2013-05-16,Thu

↑back Search

Time Nickname Message
00:14 🔗 SketchCow All right, calling it, going to keep it at 1,000
00:19 🔗 SketchCow http://teamarchive0.fnf.archive.org:8088/mrtg/committed-day.png tells the story.
00:31 🔗 hneio_ geez
00:49 🔗 underscor SketchCow: thanks for unban
00:53 🔗 underscor also, wow, we're keeping fos busy
00:57 🔗 hneio_ I'm no longer even uploading
00:57 🔗 hneio_ not one of the thousand
00:57 🔗 hneio_ luckily I still have 2 threads left which haven't finished downloading
01:00 🔗 SketchCow We're truly hosing this machine.
01:00 🔗 SketchCow I may kill rsync and try again
01:04 🔗 rabid_mar hello everyone
01:04 🔗 rabid_mar my warrior's having some issues
01:04 🔗 hneio upload problems?
01:04 🔗 rabid_mar rsync, yeah
01:04 🔗 hneio the server's under heavy load
01:04 🔗 hneio just wait
01:04 🔗 rabid_mar I figured
01:04 🔗 hneio http://teamarchive0.fnf.archive.org:8088/mrtg/committed-day.png
01:04 🔗 rabid_mar just checking in
01:04 🔗 hneio look at that
01:04 🔗 rabid_mar damn
01:05 🔗 rabid_mar that's all legitimate warrior traffic?
01:05 🔗 rabid_mar or is something else going on
01:06 🔗 hneio SketchCow ^
01:07 🔗 SketchCow Hneio, you linked rabid_mar to the memory usage of the server..
01:08 🔗 hneio yorsh
01:08 🔗 hneio that was the idea
01:09 🔗 rabid_mar I thought it was disk io
01:09 🔗 rabid_mar don't mind me
01:10 🔗 hneio that's here http://teamarchive0.fnf.archive.org:8088/mrtg/diskv3-day.png
01:13 🔗 rabid_mar I found the page with everything on it
01:13 🔗 rabid_mar out of curiosity, is this just elevated usage or is there something else going on with the server?
01:14 🔗 rabid_mar or are you guys out of the loop because this is one of archive.org's machines
01:15 🔗 SketchCow Yes
01:16 🔗 SketchCow this was a case where we had one of the members who was relatively energetic have machines pounding the machine
01:16 🔗 hneio man... 5 finished RSyncUploads
01:17 🔗 hneio only one thread left downloading
01:18 🔗 rabid_mar gotcha, thanks for the info
01:19 🔗 SketchCow 5457
01:19 🔗 SketchCow 4709
01:19 🔗 SketchCow 2372
01:20 🔗 SketchCow So good.
01:25 🔗 hneio now I'm sending incremental file list
01:25 🔗 hneio we'll see if that goes through...
13:56 🔗 Smiley Is fos still on fire?
14:01 🔗 ersi Yes. Also, the issue was discussed in #warrior and #preposterus earlier. Maybe keep it somewhere there.
14:01 🔗 * Smiley poits to the channel name
14:02 🔗 ersi Oh snap, you got me
14:02 🔗 Smiley Only other place that seems sensible is the tracker admin channel, but that's kind private :P
14:02 🔗 ersi I did make you look, though? :D
14:02 🔗 Smiley haha yes I did :D
14:02 🔗 Smiley Was wondering if my irssi wasn't redrawing correctly
14:02 🔗 ersi mwahaha! Let's say.. that was my plan all along!
15:28 🔗 SketchCow http://i.imgur.com/ERtArKf.png
15:29 🔗 SketchCow The #warrior channel is the place to discuss the warrior.
15:29 🔗 SketchCow I've got it going now, it hit 400 connections.
15:29 🔗 SketchCow Little I can do about this, I am NOT raising it right now.
15:29 🔗 SketchCow That killed the machine.
15:29 🔗 SketchCow top - 15:29:32 up 15 days, 20:10, 6 users, load average: 409.33, 391.02, 263.50
15:29 🔗 SketchCow See, it's still a little pissed now.
15:33 🔗 Smiley tbh loower it if it's an issue
15:33 🔗 Smiley the warriors can wait.
15:34 🔗 Smiley better they wait a bit than the box exploding.
19:39 🔗 SketchCow So, seriously.
19:39 🔗 SketchCow Right now we have underscor running well over a hundred processes on archive.org machines, jamming all their shit into one poor overworked public facing archive.org machine, to upload to archive.org.
19:39 🔗 SketchCow I feel like.... I feel like maybe we're adding a useless step there.
19:40 🔗 SketchCow I would like underscor and alard to talk and maybe have one of his ravenous insanity boxes play the part of relay to archive.org's machines.
19:42 🔗 underscor SketchCow: kau
19:42 🔗 underscor kay*
19:42 🔗 alard Hello.
19:42 🔗 underscor Hey alard
19:42 🔗 SketchCow Two of you. Work together.
19:43 🔗 SketchCow It'll take you 9 seconds, you're geniuses
19:43 🔗 SketchCow Because FOS does quintuple duty, if we can use an alternate machine, it works just as well.
19:43 🔗 SketchCow If it was a dedicated relay, it'd probably be happy, but it runs statusboard, I have uploads going on, I have torrents going on, and all sorts of other things.
19:44 🔗 SketchCow It can work fine as a general relay, but underscor specifically is just a tsunami of data.
19:44 🔗 SketchCow And more importantly, connections.
19:44 🔗 SketchCow He's doing 100-150 connections a second.
19:44 🔗 SketchCow I mean minute.
19:45 🔗 underscor alard: are the package-and-upload scripts easily portable?
19:45 🔗 alard https://github.com/ArchiveTeam/archiveteam-megawarc-factory
19:45 🔗 underscor I have another box(es) I can run them on to split the load with fos
19:45 🔗 alard It should be relatively easy, I think.
19:45 🔗 SketchCow Yes, use that.
19:45 🔗 alard Set up rsync, fill in the configuration file, go.
19:45 🔗 SketchCow Do it with one of your boxes, make sure it's doing the right thing, then add more to it.
19:45 🔗 SketchCow Don't just flop over, that's what a 19 year old does
19:46 🔗 SketchCow Then you're all oil and flying parts
19:46 🔗 SketchCow But I'm watching FOS snowball trying to keep up AND do the gzip/upload work
19:46 🔗 SketchCow load average: 112.89, 115.99, 116.29
19:46 🔗 alard Did you set any ionice priorities?
19:47 🔗 SketchCow Not even one.
19:47 🔗 SketchCow I could do that, it starts to get silly, though.
19:47 🔗 alard You could try to give the gzip/megawarc bits a lower priority. They'll grab everything they can.
19:47 🔗 alard It helped on short's server.
19:48 🔗 SketchCow The core issue is we have 32 internal servers slamming another internal server when they could be 31 internal servers square-dancing with a specific server.
19:48 🔗 SketchCow ionicing will work but the snowballing will still happen.
19:48 🔗 SketchCow The drive this is on is 41% full, 9tb
19:48 🔗 SketchCow It's all these two projects.
19:49 🔗 alard Nothing helps against 32 servers, no.
19:49 🔗 SketchCow And when I go to watch the logs, it's just underscor slamming the shit out of the machine.
19:49 🔗 underscor Are we doing 25 or 50gb chunks?
19:49 🔗 SketchCow 50gb.
19:49 🔗 SketchCow You've successfully made uploading robust enough for it to be on several servers, alard. We should use that.
19:49 🔗 alard I set it to 25GB for a while because some of the 50GB files weren't uploading well, but maybe that was just because of the disk error.
19:49 🔗 underscor What's the collection ID?
19:50 🔗 underscor and the item title pattern, item identifier prefix, and file prefix?
19:50 🔗 alard project?
19:50 🔗 underscor posterous
19:50 🔗 SketchCow Take it to privatew
19:51 🔗 SketchCow Just slam through it, it'll take 10 minutes.
19:51 🔗 SketchCow It's a very good script.
20:01 🔗 SketchCow I am assuming it's running.
20:01 🔗 SketchCow Guess what my hint is.
20:04 🔗 underscor FOS doing better?
20:05 🔗 underscor I'm bringing in data at 585mbps
20:05 🔗 underscor Damn
20:06 🔗 SketchCow Well, I just blocked you
20:06 🔗 SketchCow Feels good, man
20:06 🔗 underscor :c
20:07 🔗 SketchCow Having done that, I'm increasing connections to 200
20:07 🔗 underscor Total DISK WRITE: 90.77M/s
20:07 🔗 underscor Feels good, man
20:08 🔗 SketchCow Yeah, you were DDOSing the living shit fuck out of FOS
20:08 🔗 * SketchCow renames it OS
20:08 🔗 SketchCow DOS
20:08 🔗 SketchCow Denial of Solitude
20:08 🔗 underscor hahaha
20:08 🔗 underscor I was DOSing the FOS?
20:08 🔗 SketchCow HERE IN MY FORTRESS OF what the fuck are you doing
20:08 🔗 underscor :D :D :D
20:08 🔗 underscor Now I'm DOSing my own box hahaha
20:11 🔗 SketchCow It has certainly made the tracker impressive
20:19 🔗 PepsiMax Nice, the machine is happy now I guess
20:19 🔗 PepsiMax Since the uploads are coming trough :-)
20:26 🔗 Smiley half the world just saw random uploads all start
20:26 🔗 Smiley they think skynet just booted off FOS.
20:26 🔗 Smiley It's consuming all the worlds infomation!
22:26 🔗 PepsiMax Oh god it's slow again :-(
22:27 🔗 Smiley :O
22:29 🔗 * Smiley pokes SketchCow
22:30 🔗 Smiley If it's just the same issue, it'll sort it's self out
22:32 🔗 PepsiMax nah
22:33 🔗 PepsiMax ersi tried to calm me down yesterday too
22:36 🔗 Smiley PepsiMax: huh?
22:37 🔗 PepsiMax hehe
22:37 🔗 PepsiMax 20:09:34 < PepsiMax> :(:(:(:(:(
22:37 🔗 PepsiMax 20:58:52 <@ersi> PepsiMax: I'm able to upload occationally
22:37 🔗 PepsiMax 20:58:56 <@ersi> So I guess it's easing up
22:48 🔗 Smiley PepsiMax: nod.
22:55 🔗 SketchCow WHAAAAT
22:55 🔗 SketchCow We're at max connections. 200.
22:56 🔗 SketchCow Also, it's not slow, it's fast as fuck
22:56 🔗 SketchCow It's that it's got too many people who want the teat
22:56 🔗 SketchCow Now, if you want
22:56 🔗 SketchCow I can fuck the machine up REALLY good, turn it ALL the way to 11
22:56 🔗 SketchCow Then you can feel like it's "faster"
22:56 🔗 SketchCow while it goes into full melted slag
22:57 🔗 SketchCow At this point, underscor does not use the machine.
22:57 🔗 SketchCow So this is everyone else.
22:57 🔗 SketchCow Some people are uploading slow.
22:57 🔗 SketchCow If I add more, more goes slow.
22:57 🔗 SketchCow Load average is at 200
22:57 🔗 SketchCow That's lagwagon territory
22:57 🔗 SketchCow If I put it to 300, it'll fill at 300.
22:57 🔗 SketchCow Etc.
22:57 🔗 SketchCow Etc.
22:57 🔗 SketchCow Etc.
22:57 🔗 SketchCow Etc.
22:57 🔗 SketchCow Alard worked very hard for a machine to automatically just "work"
22:58 🔗 SketchCow It doesn't need you, the concerned sweaty meatbag, staring at it, hoping the magic dust will work.
22:58 🔗 SketchCow I'm doing some other stuff to it internally, it might help.
22:58 🔗 omf_ meatbag lol
22:59 🔗 Smiley lol
22:59 🔗 Smiley Is that me or PepsiMax ?
22:59 🔗 Smiley XD
22:59 🔗 SketchCow Both
22:59 🔗 SketchCow all
22:59 🔗 SketchCow Everyone
22:59 🔗 flaushy i am one of the slow uploaders^^
22:59 🔗 SketchCow people who don't even know about the project
22:59 🔗 Smiley lol
22:59 🔗 SketchCow girls in bikinis. dogs in restaurants.
22:59 🔗 SketchCow trees.
23:00 🔗 SketchCow silverware piles. mushrooms.
23:00 🔗 SketchCow Everyone gets a Paddlin'
23:00 🔗 SketchCow We're shoving 400 pounds of shit into a 5 pound bag. I'm working with it.
23:00 🔗 SketchCow But this is why we have error correction.
23:01 🔗 SketchCow The big thing is people with 56k Wondermodems opening 6 instances to 'help'
23:01 🔗 SketchCow Not much we can do about this, or eval it to stop that situation.
23:02 🔗 * Smiley glares at flaushy XD
23:02 🔗 SketchCow hilarious.
23:02 🔗 SketchCow a bucket of hilarious.
23:02 🔗 SketchCow back to movie.
23:02 🔗 Smiley as long as that's all it is, and not something else melting
23:02 🔗 SketchCow Every few minutes, the machine runs a java instance to generate a new book
23:02 🔗 SketchCow for statusboard.archive.org.
23:02 🔗 SketchCow That eats the CPU for a minute or two
23:02 🔗 SketchCow Then back
23:02 🔗 Smiley Ahhhhhh ok
23:02 🔗 SketchCow Just one of many things on there.
23:07 🔗 Howlin1 I keep getting a max connection message :/
23:08 🔗 flaushy it will work eventually
23:08 🔗 flaushy just be patient
23:09 🔗 Howlin1 True, just frustrating at times.
23:50 🔗 SketchCow underscor: Stop sending things to FOS
23:51 🔗 SketchCow You are murdering that machine.
23:51 🔗 SketchCow Even with a goddamn DENY, it's flooding the machine.
23:51 🔗 SketchCow I reboot the whole damn thing, it's JUST running rsync
23:51 🔗 SketchCow And when I turned it on, it dropped to basically nothing.
23:51 🔗 SketchCow You're killing it.
23:51 🔗 SketchCow Stop.
23:52 🔗 SketchCow Jesus, what are you doing.
23:52 🔗 SketchCow This thing is at 245 load.
23:52 🔗 SketchCow From rsync hits.
23:52 🔗 SketchCow Just rsync hits.
irclogger-viewer