#archiveteam 2015-11-02,Mon

↑back Search

Time Nickname Message
00:00 🔗 Meeh has joined #archiveteam
00:11 🔗 ersi has quit IRC (Read error: Operation timed out)
00:11 🔗 ersi_ has joined #archiveteam
00:18 🔗 antomatic has quit IRC (Read error: Operation timed out)
00:27 🔗 Meeh has quit IRC (Read error: Connection reset by peer)
00:28 🔗 Meeh has joined #archiveteam
00:40 🔗 antomatic has joined #archiveteam
00:40 🔗 swebb sets mode: +o antomatic
00:46 🔗 xk_id has quit IRC (Remote host closed the connection)
01:07 🔗 mrcave_ has joined #archiveteam
01:14 🔗 mrcave_ has quit IRC (Quit: Page closed)
01:17 🔗 Ravenloft has joined #archiveteam
01:28 🔗 MMovie1 has quit IRC (Ping timeout: 310 seconds)
01:36 🔗 MMovie has joined #archiveteam
01:37 🔗 zenguy_pc has quit IRC (Read error: Operation timed out)
01:38 🔗 primus104 has quit IRC (Leaving.)
01:38 🔗 Coderjoe has quit IRC (Ping timeout: 255 seconds)
01:42 🔗 nertzy has joined #archiveteam
01:46 🔗 Coderjoe has joined #archiveteam
01:51 🔗 zenguy_pc has joined #archiveteam
02:17 🔗 xk_id has joined #archiveteam
02:28 🔗 xk_id has quit IRC (Ping timeout: 615 seconds)
02:29 🔗 phuzion Do we have all of the projects that are reliant on FOS paused right now? rsync is failing for wikis and gamefront for me.
02:37 🔗 JesseW has quit IRC (Leaving.)
02:53 🔗 RichardG has quit IRC (Read error: Connection reset by peer)
02:58 🔗 RichardG has joined #archiveteam
03:08 🔗 JesseW has joined #archiveteam
03:18 🔗 xk_id has joined #archiveteam
03:26 🔗 Start has quit IRC (Quit: Disconnected.)
03:29 🔗 Start has joined #archiveteam
03:33 🔗 xk_id has quit IRC (Ping timeout: 615 seconds)
03:38 🔗 zenguy_pc has quit IRC (Read error: Operation timed out)
03:52 🔗 zenguy_pc has joined #archiveteam
03:53 🔗 Ymgve has quit IRC (Ping timeout: 506 seconds)
03:56 🔗 Meeh has quit IRC (Ping timeout: 506 seconds)
03:58 🔗 Meeh has joined #archiveteam
04:00 🔗 Ymgve has joined #archiveteam
04:08 🔗 Atluxity They seem to succeed now and again
04:12 🔗 MrRadar Does it delete stuff as it finishes uploading to IA? If so that could be why
04:17 🔗 phuzion MrRadar: If I'm not mistaken, we have a handful of projects that are rather large that are consuming a good chunk of the space on FOS right now
04:18 🔗 phuzion thingiverse and blingee takes up like 7TB as is
04:18 🔗 phuzion Gamefront is 10TB
04:21 🔗 phuzion I have no idea how big FOS is in terms of disk space, but I thought there was some megawarc process that happened on FOS before it got pushed to IA
04:23 🔗 aaaaaaaaa has quit IRC (Leaving)
04:24 🔗 xk_id has joined #archiveteam
04:36 🔗 xk_id has quit IRC (Ping timeout: 614 seconds)
04:37 🔗 yipdw it's not a space problem
04:38 🔗 yipdw $ w
04:38 🔗 yipdw 04:37:38 up 136 days, 8:15, 1 user, load average: 53.65, 54.40, 55.92
04:38 🔗 yipdw there's ~5 TB free on data target partitions
04:39 🔗 yipdw I don't know what error you're seeing, but if it's max connections reached or something related to load that's why
04:39 🔗 MrRadar Wow, impressive load average
04:39 🔗 yipdw the remedy is forcing people to back off
04:39 🔗 yipdw which I will do now
04:39 🔗 yipdw something tragedy of the commons something
04:42 🔗 yipdw there's also a lot of stuck rsync process messages in dmesg blocked on what looks like I/O https://gist.github.com/yipdw/a60141933eb741d5014d
04:43 🔗 yipdw at the moment, there are two megawarc packers active; each packer is very I/O intensive
04:53 🔗 phuzion yipdw: Ah, ok. Out of curiosity, how much space does FOS have in total?
04:54 🔗 yipdw 16.2 TB
04:55 🔗 yipdw I don't know how much of that is available for this stuff
04:55 🔗 phuzion Gotcha.
04:56 🔗 phuzion my rsync jobs are running again
04:56 🔗 phuzion just so you know
04:56 🔗 yipdw you may see increased rates of max connections reached errors
04:56 🔗 yipdw there may be (probably is) someone running a ludicrous number of works
04:56 🔗 yipdw workers
04:57 🔗 phuzion Are you able to log by IP or something?
04:57 🔗 yipdw the tracker does yeah
04:57 🔗 phuzion Rather, do those logs exist?
04:57 🔗 yipdw also by username
04:57 🔗 phuzion ok cool
04:57 🔗 Atluxity yipdw: I have 200 instances on wikis, then 200 on gamefront...
04:58 🔗 yipdw well there we go
04:58 🔗 phuzion Are we still waiting on any of the data to come back from the telenor grab? It could be HCross with the scaleway servers he was running
05:00 🔗 Atluxity let me know if you want me to change my setup
05:00 🔗 Atluxity I have 20 servers, each with its own ip, running 20 instanses each
05:00 🔗 phuzion Atluxity: you're running the wikiteam pipeline right?
05:00 🔗 yipdw it may be possible to adjust fos' I/O scheduling to better handle this sort of load, but I don't want to do that because fos is (supposedly) under FAI control
05:00 🔗 Atluxity phuzion: yes
05:01 🔗 yipdw and I don't know anything about the FAI loadout
05:01 🔗 phuzion Atluxity: do you get lots of 410 responses for Wikia sites?
05:01 🔗 phuzion FAI?
05:01 🔗 yipdw Fully Automated Installation
05:01 🔗 yipdw I guess I could investigate runtime tweaks, the Linux I/O scheduler can be tuned that way
05:01 🔗 Atluxity phuzion: jupp
05:01 🔗 phuzion Atluxity: ok, just wanted to make sure I wasn't going crazy
05:02 🔗 phuzion or getting my IP banned or something
05:02 🔗 phuzion yipdw: oh, it's config management, right?
05:02 🔗 yipdw yeah
05:02 🔗 phuzion ok
05:02 🔗 yipdw oh huh, sar is also installed, neat
05:03 🔗 yipdw huh 1200 transfers/second, I don't know where that is on the what-the-hardware-can-do scale
05:04 🔗 phuzion yipdw: Process RsyncUpload returned exit code 5 for Item file:2264798
05:04 🔗 phuzion Is that "too many connections to the box" or what?
05:04 🔗 yipdw there's an error message that states "max connections reached"
05:04 🔗 yipdw it's printed out
05:04 🔗 yipdw exit code 5 from rsync means "error starting client-server protocol"
05:04 🔗 phuzion God damn
05:04 🔗 phuzion I really ought to read the entire output for once
05:04 🔗 phuzion lol
05:05 🔗 yipdw I do recommend people lower their worker count, fos is a bottleneck
05:05 🔗 yipdw and it's at max connections 50 anyway
05:05 🔗 Atluxity ok
05:05 🔗 Atluxity I recognize myself as people
05:05 🔗 yipdw to scale higher we need more rsync targets
05:06 🔗 phuzion yipdw: FOS runs debian, right? Jessie?
05:07 🔗 yipdw Ubuntu
05:07 🔗 yipdw or at least that's what /etc/issue says
05:07 🔗 yipdw it has apt so I guess that's close enough
05:07 🔗 phuzion Ah, ok. I'm gonna see if there's any optimizations that can be made on the rsync config. Do you know if rsync is using the distro default config?
05:08 🔗 bzc6p_ has joined #archiveteam
05:08 🔗 swebb sets mode: +o bzc6p_
05:08 🔗 yipdw er wait
05:08 🔗 yipdw argh that /etc/issue was my own system
05:08 🔗 yipdw where the hell is my fos terminal
05:08 🔗 Atluxity I'll stop my workers, then set them up with new instance count
05:08 🔗 yipdw well it's still Ubuntu heh
05:09 🔗 yipdw phuzion: yes, aside from module configurations, I don't see anything custom in rsyncd.conf
05:09 🔗 bzc6p has quit IRC (Read error: Operation timed out)
05:09 🔗 PepsiMax_ has joined #archiveteam
05:09 🔗 Atluxity any idea what will be a more sane instance count?
05:09 🔗 phuzion ok, I'll see if there's anything I can do to optimize rsyncd to accept more connections to reduce this bottleneck.
05:09 🔗 yipdw well, we can accept more connections fine
05:10 🔗 yipdw the I/O subsystem can't keep up
05:10 🔗 phuzion Ohhh
05:10 🔗 yipdw Atluxity: I'm not sure, perhaps start low and scale slowly
05:10 🔗 phuzion I thought rsync was rejecting the connection because it couldn't open a connection for some reason
05:10 🔗 dcmorton has quit IRC (Read error: Operation timed out)
05:11 🔗 dserodio has quit IRC (Read error: Operation timed out)
05:11 🔗 HarryCros has quit IRC (Read error: Operation timed out)
05:11 🔗 yipdw yeah, if you saw max connections reached, it's because the max connection count for that module was hit
05:11 🔗 Sanqui has quit IRC (Ping timeout: 499 seconds)
05:11 🔗 yipdw but I set that explicitly because at the old limit fos was getting killed
05:11 🔗 phuzion Ohhhhhhh, so the limit is placed in rsyncd artificially so we don't murder IO ok
05:11 🔗 yipdw old limit was 150; we're at 25 now (50 still didn't seem to help)
05:12 🔗 robink has quit IRC (Ping timeout: 499 seconds)
05:13 🔗 phuzion I assume the IO isn't due to sheer write speed, it's the insane number of transactions we're putting on the disk, right?
05:13 🔗 yipdw I don't know
05:13 🔗 yipdw I do know that there are two gamefront megawarc packers running
05:13 🔗 yipdw gamefront is videos, right
05:13 🔗 phuzion I'm not sure.
05:14 🔗 phuzion I've seen items anywhere between 0.1MB to 3-500MB
05:14 🔗 phuzion 562MB just hit on the tracker
05:15 🔗 phuzion yipdw: If you had to guess, how long would those megawarc packers take to run?
05:15 🔗 PepsiMax has quit IRC (Ping timeout: 730 seconds)
05:15 🔗 phuzion Also, how much data is being packed?
05:15 🔗 chazchaz has quit IRC (Ping timeout: 369 seconds)
05:16 🔗 yipdw don't know
05:16 🔗 atlogbot has quit IRC (Ping timeout: 369 seconds)
05:16 🔗 phuzion I'm wondering if putting a couple of 512GB SSDs in FOS would help for megawarc packing
05:16 🔗 yipdw probably
05:16 🔗 yipdw maintenance cost goes up though
05:17 🔗 Atluxity I'll ask slappfisk to back down a bit too... he also has access to many servers
05:17 🔗 phuzion I'd bet we could crowdfund $500 for 2x512GB SSDs pretty easily. I'd probably chip in $50 or something
05:18 🔗 phuzion If that would be anywhere close to enough storage for megawarc packing, that is.
05:18 🔗 yipdw it's not the initial thing that worries me, it's upkeep
05:18 🔗 yipdw SSDs are gonna die
05:19 🔗 phuzion True.
05:19 🔗 yipdw and all that upkeep goes into IA's budget
05:19 🔗 phuzion Right. Maybe some 15K SAS drives then :)
05:19 🔗 yipdw unless we continue that funding thing as a recurring donation or something
05:19 🔗 phuzion Less likely to be murdered by IO over time.
05:20 🔗 phuzion But also significantly more expensive.
05:21 🔗 kniffy has quit IRC (Ping timeout: 369 seconds)
05:22 🔗 phuzion yipdw: Perhaps a stupid question, but how is the storage on FOS split up? One big mount or multiple smaller mounts?
05:23 🔗 yipdw two RAID devices, each with their own mount
05:23 🔗 phuzion Are megawarcs packed on one and incoming files for rsync on another?
05:24 🔗 phuzion Or is it just "here's bin A, here's bin B, do whatever on either"
05:25 🔗 yipdw megawarc chunking and packing need to be done on the same filesystem, packing -> uploading can move elsewhere
05:25 🔗 yipdw or rather https://github.com/ArchiveTeam/archiveteam-megawarc-factory#filesystems explains it better
05:26 🔗 kniffy has joined #archiveteam
05:26 🔗 phuzion Ok, that makes sense.
05:28 🔗 yipdw I'm not sure which filesystem the packer is on
05:28 🔗 yipdw yet
05:29 🔗 yipdw I think it's on filesystem 0, all rsync goes to filesystem 1
05:29 🔗 yipdw and I'm not sure the chunker is running
05:29 🔗 yipdw it doesn't look like it
05:29 🔗 yipdw oh wait it's not the same chunker I know
05:29 🔗 phuzion pgrep chunker doesn't return anything on the system?
05:29 🔗 yipdw interesting
05:29 🔗 phuzion oh
05:30 🔗 yipdw I am uh not going to touch that
05:32 🔗 kniffy has quit IRC (Quit: :^))
05:33 🔗 kniffy has joined #archiveteam
05:37 🔗 dan- yipdw: heyo, have some stuff to ask about ArchiveBot, mind if I PM you?
05:39 🔗 atlogbot has joined #archiveteam
05:39 🔗 HarryCros has joined #archiveteam
05:40 🔗 chazchaz has joined #archiveteam
05:40 🔗 zenguy_pc has quit IRC (Read error: Operation timed out)
05:41 🔗 dserodio has joined #archiveteam
05:41 🔗 dcmorton has joined #archiveteam
05:42 🔗 kniffy has quit IRC (Quit: :^))
05:43 🔗 kniffy has joined #archiveteam
05:43 🔗 kniffy has quit IRC (Remote host closed the connection)
05:43 🔗 kniffy has joined #archiveteam
05:53 🔗 zenguy_pc has joined #archiveteam
05:53 🔗 xk_id has joined #archiveteam
05:54 🔗 dcmorton has quit IRC (Read error: Operation timed out)
05:54 🔗 dserodio has quit IRC (Read error: Operation timed out)
05:55 🔗 atlogbot has quit IRC (Read error: Operation timed out)
05:56 🔗 HarryCros has quit IRC (Read error: Operation timed out)
05:57 🔗 WinterFox has joined #archiveteam
05:57 🔗 kniffy has quit IRC (Quit: :^))
05:57 🔗 kniffy has joined #archiveteam
06:00 🔗 chazchaz has quit IRC (Ping timeout: 369 seconds)
06:04 🔗 Sanqui has joined #archiveteam
06:04 🔗 slyphic has quit IRC (Read error: Operation timed out)
06:05 🔗 Deewiant has joined #archiveteam
06:06 🔗 HarryCros has joined #archiveteam
06:06 🔗 atlogbot has joined #archiveteam
06:06 🔗 chazchaz has joined #archiveteam
06:07 🔗 slyphic has joined #archiveteam
06:08 🔗 yipdw dan-: yeah sure that's cool
06:08 🔗 dserodio has joined #archiveteam
06:08 🔗 dcmorton has joined #archiveteam
06:25 🔗 cvb has joined #archiveteam
06:33 🔗 Sanqui has quit IRC (Ping timeout: 499 seconds)
06:37 🔗 cvb has quit IRC (Read error: Operation timed out)
06:38 🔗 cvb has joined #archiveteam
06:38 🔗 HarryCros has quit IRC (Read error: Operation timed out)
06:41 🔗 chazchaz has quit IRC (Read error: Operation timed out)
06:41 🔗 atlogbot has quit IRC (Ping timeout: 369 seconds)
06:43 🔗 dserodio has quit IRC (Read error: Operation timed out)
06:44 🔗 slyphic has quit IRC (Read error: Operation timed out)
06:49 🔗 primus104 has joined #archiveteam
06:54 🔗 xk_id has quit IRC (Remote host closed the connection)
06:54 🔗 chazchaz has joined #archiveteam
06:55 🔗 dcmorton has quit IRC (Ping timeout: 369 seconds)
06:55 🔗 Atluxity there... backing down from 2 x 10 x 20 to 2 x 10 x 5 seems to have helped the flow of items quite a bit
07:00 🔗 JesseW Atluxity: feel free to toss the others at urlteam, I think. :-)
07:01 🔗 cvb has quit IRC (Quit: Leaving)
07:02 🔗 phuzion JesseW: I'm fairly certain URLteam has enough workers on it, but more certainly can't hurt in the long run.
07:02 🔗 Atluxity JesseW: got 200 instances at urlteam already :)
07:02 🔗 phuzion Jesus
07:03 🔗 Atluxity but I did not notice that it increased scans per second, so I dont know what the bottleneck is
07:03 🔗 phuzion Atluxity: Who do you host all this with?
07:03 🔗 Atluxity phuzion: I am testing a new openstack rig at work
07:04 🔗 phuzion Ahhh, jealous
07:04 🔗 yipdw oh jesus
07:04 🔗 Atluxity I could do whatever, just as long as I kept under 1Gbit\s and did not expect anything to stay alive or be kept safe
07:05 🔗 phuzion Haha christ, that's insane. "Here's some crazy new gear to mess around with. Keep the network traffic under 1gbps. Have fun."
07:05 🔗 Atluxity so I dont think this will be a nice place for archivebot instance :\
07:06 🔗 phuzion :(
07:06 🔗 dserodio has joined #archiveteam
07:06 🔗 dcmorton has joined #archiveteam
07:07 🔗 atlogbot has joined #archiveteam
07:08 🔗 HarryCros has joined #archiveteam
07:08 🔗 Jordan_ yipdw: Mind if I pm you? I had a couple questions if you have time
07:09 🔗 slyphic has joined #archiveteam
07:09 🔗 yipdw Jordan_: sure, that's cool
07:09 🔗 Kenshin has quit IRC (Read error: Operation timed out)
07:10 🔗 Atluxity phuzion: my deploy-script is a titanpad piped to shell as root.... *giggles*
07:10 🔗 phuzion you monster
07:10 🔗 phuzion lol
07:10 🔗 Atluxity fun to have servers one does not care about
07:13 🔗 Atluxity holy crap, gamefront got 59K items out
07:13 🔗 Kenshin has joined #archiveteam
07:14 🔗 phuzion Yeah, we might wanna see about getting some of the older items re-queued
07:17 🔗 JesseW who are these random new (at least, new to me) people with private questions for yipdw ...?
07:19 🔗 Atluxity they seems to scare to ask directly?
07:20 🔗 Atluxity Jordan: randomly several people have asked yipdw about privmsg for questions in a short time. Is something going on?
07:20 🔗 Atluxity Jordan_ even
07:20 🔗 yipdw https://www.youtube.com/watch?v=xxMb_8ZpjIY
07:21 🔗 slyphic has quit IRC (Read error: Operation timed out)
07:21 🔗 primus104 has quit IRC (Leaving.)
07:21 🔗 JesseW must be it. :-)
07:21 🔗 yipdw they're ArchiveBot-related questions
07:22 🔗 dserodio has quit IRC (Read error: Operation timed out)
07:23 🔗 dcmorton has quit IRC (Read error: Operation timed out)
07:23 🔗 JesseW just seems unusual to have two people ask to ask in quick succession
07:23 🔗 JesseW but, not my business, so <shrug>
07:24 🔗 HarryCros has quit IRC (Read error: Operation timed out)
07:27 🔗 phuzion yipdw: is there a way to view the queue on archivebot?
07:27 🔗 yipdw !pending
07:27 🔗 phuzion Web interface?
07:28 🔗 yipdw oh, no
07:28 🔗 phuzion ok
07:28 🔗 phuzion was just curious
07:34 🔗 phuzion Jeez, there's jobs running since June?!?
07:35 🔗 atlogbot has quit IRC (Ping timeout: 369 seconds)
07:37 🔗 zenguy_pc has quit IRC (Read error: Operation timed out)
07:38 🔗 phuzion Question about the Gamefront grab. Is it normal to have dozens or hundreds of hits to facebook when running an item for that job?
07:41 🔗 atlogbot has joined #archiveteam
07:41 🔗 jleclanch has quit IRC (Read error: Connection reset by peer)
07:42 🔗 slyphic has joined #archiveteam
07:42 🔗 HarryCros has joined #archiveteam
07:42 🔗 jleclanch has joined #archiveteam
07:43 🔗 dcmorton has joined #archiveteam
07:44 🔗 dserodio has joined #archiveteam
07:47 🔗 Atluxity I hope so
07:47 🔗 Atluxity I see alot of that
07:51 🔗 phuzion If it isn't, we could probably speed up the retrieval of items by blocking those.
07:53 🔗 zenguy_pc has joined #archiveteam
07:55 🔗 jleclanch has quit IRC (Read error: Connection reset by peer)
07:57 🔗 jleclanch has joined #archiveteam
08:01 🔗 Atluxity yeah
08:07 🔗 jleclanch has quit IRC (Read error: Connection reset by peer)
08:09 🔗 jleclanch has joined #archiveteam
08:11 🔗 JesseW has quit IRC (Leaving.)
08:16 🔗 jleclanch has quit IRC (Read error: Connection reset by peer)
08:18 🔗 jleclanch has joined #archiveteam
08:18 🔗 jleclanch has quit IRC (Remote host closed the connection)
08:25 🔗 xk_id has joined #archiveteam
08:30 🔗 jleclanch has joined #archiveteam
08:34 🔗 xk_id has quit IRC (Read error: Operation timed out)
08:36 🔗 jleclanch has quit IRC (Read error: Operation timed out)
08:39 🔗 jleclanch has joined #archiveteam
08:50 🔗 jleclanch has quit IRC (Remote host closed the connection)
08:55 🔗 primus104 has joined #archiveteam
09:05 🔗 ersi_ is now known as ersi
09:11 🔗 atomotic has joined #archiveteam
09:31 🔗 Stiletto has quit IRC (Ping timeout: 272 seconds)
09:32 🔗 Stiletto has joined #archiveteam
09:34 🔗 schbirid has joined #archiveteam
09:41 🔗 zenguy_pc has quit IRC (Read error: Operation timed out)
09:49 🔗 scyther has joined #archiveteam
09:50 🔗 Guest7171 has joined #archiveteam
09:51 🔗 arkiver yipdw: FOS was running for weeks with 1 or 2 megawarc packers running and uploading and 150 rsync connections. Now it suddenly can't handle it anymore?
09:52 🔗 arkiver The issue in the wikis grab with status code 410 was fixed a few days ago, I forgot to set the new version in the tracker. Will do that now
09:52 🔗 arkiver Yesterday actually ^
09:53 🔗 zenguy_pc has joined #archiveteam
10:03 🔗 primus104 has quit IRC (Leaving.)
10:06 🔗 Atluxity sigh
10:08 🔗 Atluxity starting stopping this train is not as easy as I would like... but... :P
10:08 🔗 Atluxity oh wait...
10:08 🔗 Atluxity arkiver: so if I have pulled the git repo last 24 hours then I have the new version and dont need to do anything?
10:09 🔗 arkiver If you don't see any message that you should update your scripts you're fine
10:10 🔗 arkiver Are you able to bring all the concurrent you had online back up if needed?
10:10 🔗 arkiver (not now yet)
10:10 🔗 Atluxity yeah
10:10 🔗 arkiver ok, awesome
10:10 🔗 Atluxity I think they could do more concurrent too, the load was not high
10:14 🔗 arkiver sounds good
10:15 🔗 arkiver when we figure out what's going on with FOS or have a new rsync target we might need them :)
10:19 🔗 midas1 is now known as midas
10:27 🔗 xk_id has joined #archiveteam
10:37 🔗 Atluxity going to test to stop all my work to see what the impact is in general
10:37 🔗 Atluxity spent some effort making it easier to stop and start stuff
10:39 🔗 xk_id has quit IRC (Ping timeout: 615 seconds)
10:42 🔗 jleclanch has joined #archiveteam
10:46 🔗 scyther_ has joined #archiveteam
10:53 🔗 scyther has quit IRC (Read error: Operation timed out)
10:54 🔗 Muad-Dib has joined #archiveteam
11:20 🔗 Guest7171 has quit IRC (Quit: Textual IRC Client: www.textualapp.com)
11:29 🔗 xk_id has joined #archiveteam
11:37 🔗 xk_id has quit IRC (Read error: Operation timed out)
11:40 🔗 is- has quit IRC (Read error: Operation timed out)
11:41 🔗 is- has joined #archiveteam
11:42 🔗 zenguy_pc has quit IRC (Read error: Operation timed out)
11:54 🔗 zenguy_pc has joined #archiveteam
11:58 🔗 vitzli has joined #archiveteam
12:11 🔗 WinterFox has quit IRC (Remote host closed the connection)
12:28 🔗 atomotic has quit IRC (Quit: Textual IRC Client: www.textualapp.com)
12:30 🔗 xk_id has joined #archiveteam
12:41 🔗 xk_id has quit IRC (Ping timeout: 615 seconds)
12:57 🔗 BlueMaxim has quit IRC (Quit: Leaving)
13:03 🔗 rduser has quit IRC (Read error: Operation timed out)
13:03 🔗 scyther_ has quit IRC (Ping timeout: 272 seconds)
13:17 🔗 HCross I do have a FOS suggestion. EU FOS?
13:20 🔗 arkiver2 has joined #archiveteam
13:23 🔗 khaoohs has quit IRC (Read error: Operation timed out)
13:31 🔗 xk_id has joined #archiveteam
13:31 🔗 ersi Uh, no?
13:38 🔗 phuzion arkiver: thanks for fixing that 410 error :)
13:41 🔗 phuzion I've got 20 threads on the wiki job now
13:42 🔗 xk_id has quit IRC (Ping timeout: 615 seconds)
13:47 🔗 Atluxity ersi: why?
13:49 🔗 jspiros has joined #archiveteam
13:49 🔗 ersi Isn't the sole point of FOS that it's in IA's network? and that it got storage to buffer for a while?
13:53 🔗 phuzion ersi: the idea of FOS is that it's a staging server that can take in a ton of data while it's being collected by the various scripts/warriors. Its location doesn't really matter, as long as there's suitable bandwidth between FOS and IA. The fact that it's on IA's network is convenient though, because it means that there's no need to pay for bandwidth twice to transfer said data.
13:54 🔗 phuzion Right now, FOS's bottleneck isn't based on geographic constraints, it's I/O bound at the moment.
13:54 🔗 ersi How redundant. Thanks for basically saying the same thing, but phrasing it differently. :)
13:54 🔗 phuzion So, I'm not sure what good a FOS in EU would serve?
13:56 🔗 * ersi nods agreeingly
13:57 🔗 midas biggest issue is getting stuff from fos in IA anyway
13:57 🔗 midas or from any host to IA for that matter :p
13:57 🔗 arkiver2 has quit IRC (Ping timeout: 252 seconds)
14:01 🔗 Sanqui has joined #archiveteam
14:08 🔗 arkiver2 has joined #archiveteam
14:21 🔗 primus104 has joined #archiveteam
14:21 🔗 lytv has quit IRC (Read error: Operation timed out)
14:22 🔗 lytv has joined #archiveteam
14:30 🔗 arkiver2 has quit IRC (Ping timeout: 252 seconds)
14:33 🔗 xk_id has joined #archiveteam
14:36 🔗 nertzy has quit IRC (Quit: This computer has gone to sleep)
14:41 🔗 xk_id has quit IRC (Read error: Connection reset by peer)
14:41 🔗 xk_id has joined #archiveteam
14:52 🔗 xk_id has quit IRC (Ping timeout: 615 seconds)
15:02 🔗 arkiver2 has joined #archiveteam
15:11 🔗 xk_id has joined #archiveteam
15:13 🔗 arkiver2 has quit IRC (Ping timeout: 252 seconds)
15:17 🔗 rduser has joined #archiveteam
15:17 🔗 Start has quit IRC (Quit: Disconnected.)
15:26 🔗 atomotic has joined #archiveteam
15:38 🔗 cvb has joined #archiveteam
15:42 🔗 Start has joined #archiveteam
15:42 🔗 zenguy_pc has quit IRC (Read error: Operation timed out)
15:48 🔗 atomotic has quit IRC (Quit: Textual IRC Client: www.textualapp.com)
15:54 🔗 vitzli has quit IRC (Quit: Leaving)
15:55 🔗 zenguy_pc has joined #archiveteam
15:55 🔗 Atluxity maybe IA could start a europe-branch?
15:55 🔗 Atluxity then it could have a europe-FOS
16:04 🔗 zenguy_pc has quit IRC (Ping timeout: 310 seconds)
16:12 🔗 primus104 has quit IRC (Leaving.)
16:16 🔗 zenguy_pc has joined #archiveteam
16:21 🔗 Ghost_of_ has joined #archiveteam
16:24 🔗 phuzion Atluxity: I'm not quite sure I'm following the need for a european FOS. Are we seeing any kind of performance issues that would be resolved by having a staging target geographically closer to some of the script runners?
16:29 🔗 ohhdemgir has joined #archiveteam
16:31 🔗 MMovie has quit IRC (Ping timeout: 310 seconds)
16:34 🔗 joepie91 Atluxity: I don't believe the bottleneck exists on a network level
16:35 🔗 HCross its an IO issue
16:41 🔗 MMovie has joined #archiveteam
16:42 🔗 HCross has left Leaving
16:44 🔗 yipdw arkiver: I just report what I see, you can take it up with w and sar
16:45 🔗 phuzion Legit question, what's the bare minimum size we could get away with using as a separate rsync target? Say smaller jobs. 2TB? 5? 10?
16:45 🔗 MMovie1 has joined #archiveteam
16:47 🔗 ersi Depends on the project..
16:48 🔗 MMovie has quit IRC (Ping timeout: 310 seconds)
16:50 🔗 phuzion Obviously a 2TB FOS wouldn't be any good for Gamefront, but say telenor? That was 237GB. I'm wondering if it would be worth it to grab a 2TB Kimsufi server or something and offer it as a secondary staging server
17:01 🔗 JesseW has joined #archiveteam
17:02 🔗 atomotic has joined #archiveteam
17:02 🔗 marvinw_ has quit IRC (Read error: Operation timed out)
17:04 🔗 i0npulse has quit IRC (Ping timeout: 506 seconds)
17:10 🔗 Start has quit IRC (Read error: Operation timed out)
17:11 🔗 scyther has joined #archiveteam
17:12 🔗 Start has joined #archiveteam
17:15 🔗 JesseW has quit IRC (Leaving.)
17:16 🔗 bzc6p__ has joined #archiveteam
17:16 🔗 swebb sets mode: +o bzc6p__
17:18 🔗 bzc6p_ has quit IRC (Read error: Operation timed out)
17:34 🔗 primus104 has joined #archiveteam
17:39 🔗 arkiver2 has joined #archiveteam
17:48 🔗 arkiver2 has quit IRC (Ping timeout: 252 seconds)
17:51 🔗 riz has quit IRC (Remote host closed the connection)
17:52 🔗 riz has joined #archiveteam
18:04 🔗 zenguy_pc has quit IRC (Ping timeout: 310 seconds)
18:09 🔗 joepie91 phuzion: mind that kimsufi's are single disk
18:09 🔗 joepie91 phuzion: has a tendency of murdering your I/O
18:09 🔗 joepie91 for projects with small files
18:09 🔗 joepie91 because too many random writes
18:09 🔗 yipdw phuzion: only if you're willing to keep it online
18:10 🔗 yipdw IME the problem is rarely related to technical specifications
18:12 🔗 yipdw like, the really cool thing about fos is that it's been around long enough that we can be reasonably sure it's going to be around. this goes also for the amazing stuff Kenshin has thrown in
18:12 🔗 yipdw one-off rsyncs are fine, though they can be a bit of a pain to wrangle
18:16 🔗 zenguy_pc has joined #archiveteam
18:34 🔗 jspiros has quit IRC (Ping timeout: 186 seconds)
18:36 🔗 Start has quit IRC (Quit: Disconnected.)
18:42 🔗 joepie91 has quit IRC (Read error: Operation timed out)
18:44 🔗 Ghost_of_ has quit IRC (Quit: Leaving)
18:44 🔗 joepie91 has joined #archiveteam
18:48 🔗 aaaaaaaaa has joined #archiveteam
18:48 🔗 swebb sets mode: +o aaaaaaaaa
18:51 🔗 aaaaaaaaa has quit IRC (Client Quit)
18:57 🔗 atomotic_ has joined #archiveteam
18:59 🔗 atomotic has quit IRC (Ping timeout: 252 seconds)
19:02 🔗 jspiros has joined #archiveteam
19:11 🔗 Start has joined #archiveteam
19:16 🔗 nertzy has joined #archiveteam
19:19 🔗 Start has quit IRC (Quit: Disconnected.)
19:35 🔗 nertzy has quit IRC (Quit: This computer has gone to sleep)
19:41 🔗 vOYtEC has quit IRC (Read error: Connection reset by peer)
19:48 🔗 insane_al has joined #archiveteam
19:52 🔗 SimpBrain has quit IRC (Leaving)
19:54 🔗 aaaaaaaaa has joined #archiveteam
19:54 🔗 swebb sets mode: +o aaaaaaaaa
20:05 🔗 zenguy_pc has quit IRC (Ping timeout: 310 seconds)
20:17 🔗 zenguy_pc has joined #archiveteam
20:33 🔗 aaaaaaaa_ has joined #archiveteam
20:33 🔗 aaaaaaaaa has quit IRC (Read error: Connection reset by peer)
20:33 🔗 swebb sets mode: +o aaaaaaaa_
20:34 🔗 aaaaaaaa_ is now known as aaaaaaaaa
20:35 🔗 SketchCow Sold out largest Archive Team meeting for a BURGER
20:41 🔗 Ghost_of_ has joined #archiveteam
20:43 🔗 bzc6p__ is now known as bzc6p
20:45 🔗 SketchCow FOR. A. BURGER.
20:45 🔗 aaaaaaaaa has quit IRC (Read error: Connection reset by peer)
20:45 🔗 SketchCow Wow, people had lots of FOS talk while I'm gone.
20:46 🔗 aaaaaaaaa has joined #archiveteam
20:46 🔗 swebb sets mode: +o aaaaaaaaa
20:46 🔗 Atluxity indeed :)
20:46 🔗 phuzion SketchCow: Yeah, I'm not sure how I can be helpful short of running scripts, so I'm trying to think of solutions to problems as I see them :)
20:47 🔗 aaaaaaaaa has quit IRC (Client Quit)
20:47 🔗 Atluxity some ramlbing about euro-FOS, think everyone agreed that can be ignored
20:47 🔗 SketchCow A lot of it was "I'm not on the machine, I don't know the stats, I can't really come up with coherent solutions without that information, but WHY LET THAT STOP ME NOW" <30 lines>
20:48 🔗 Atluxity cause solving problems are fun! who cares if they are there or not :D
20:48 🔗 phuzion lol
20:49 🔗 SketchCow Things worth noting:
20:49 🔗 SketchCow 1. FOS is a VM, not its own server
20:49 🔗 SketchCow 2. Some things, the scripts are running and do the work. Some I do manually.
20:49 🔗 SketchCow 3. I was in the country of Europe all last week
20:49 🔗 SketchCow 4. Since as we know, the vast social net built into their pinko government systems means free blowjobs forever, I was busy
20:50 🔗 * ersi shrugs
20:50 🔗 SketchCow 5. I didn't know about the disk full until Arkiver told me, and I've been fighting that back
20:50 🔗 ersi Country of Europe
20:50 🔗 * ersi snickers
20:50 🔗 SketchCow 6. I suspect that there have been some notable jobs added which have no "pack and put into IA" process attached yet.,
20:52 🔗 zenguy_pc has quit IRC (Ping timeout: 310 seconds)
20:52 🔗 SketchCow So I'm quickly giving FOS the once over. I also restarted rsync in case you maniancs broke its heart or something.
20:53 🔗 Atluxity I might have been a bit eager
20:55 🔗 Start has joined #archiveteam
20:55 🔗 joepie91 SketchCow: where was this mythical burger bar that you spoke of
20:55 🔗 SketchCow Out the theater door, to the left, 5 doors down.
20:56 🔗 atomotic_ has quit IRC (Quit: Textual IRC Client: www.textualapp.com)
20:56 🔗 joepie91 SketchCow: that's the direction in which we walked, but we failed to observe a burger bar...
20:56 🔗 joepie91 then again, we had to catch a train anyway :P
20:56 🔗 SketchCow It was RIGHT there
20:56 🔗 SketchCow Schoutenstraat 15, 3512 GA Utrecht
20:57 🔗 SketchCow The nice Brazilian Hagrid found us
20:57 🔗 SketchCow Wrong address I gave you.
20:58 🔗 SketchCow Anyway, it was nice.
20:58 🔗 SketchCow Then I went to bed, got up nice and early, and then 9 hour flight to here.
20:59 🔗 joepie91 SketchCow: is it bad that I immediately know who you are refering to with "Brazilian Hagrid"
20:59 🔗 joepie91 lol
21:04 🔗 zenguy_pc has joined #archiveteam
21:05 🔗 matthusby FYI: Getting strange rsync errors randomly on gamefront, http://pastebin.com/iGxWUiq9
21:06 🔗 bzc6p phuzion: Regarding ArchiveBot queue web interface, we have this, quite nice: http://archivebot.at.ninjawedding.org:4567/
21:07 🔗 Atluxity matthusby: not seen that variant before
21:08 🔗 Atluxity today, it seems to me that the pipelines are set up to grab one item, then rsync it to some stageing place (FOS)
21:08 🔗 bzc6p xmc: I miss the fancy graphs of projects' performance on http://zeppelin.xrtc.net. Not updated since July 28. If it just needs a restart, it would be nice to have it. Thanks in advance.
21:09 🔗 Atluxity it seems to me like an rsync session is better for large transfers, and that small ones are not optimal, could perhaps the pipeline rsync after 5 items or after X items instead? would that solve anything?
21:10 🔗 phuzion Atluxity: the limit on simultaneous rsync connections is artificially imposed to keep the I/O from being killed on FOS
21:11 🔗 Atluxity ah, ok
21:12 🔗 phuzion Check logs starting about 23:15 Eastern last night, we discussed the rsync limits and whatnot
21:12 🔗 schbirid has quit IRC (Quit: Leaving)
21:17 🔗 aaaaaaaaa has joined #archiveteam
21:17 🔗 swebb sets mode: +o aaaaaaaaa
21:22 🔗 Atluxity g'nite
21:25 🔗 SketchCow SOMETHING got better on IO
21:25 🔗 SketchCow Because FOS seems (seems) zippy again
21:31 🔗 MrRadar What's the load average? it was up in the 50s last night
21:33 🔗 SketchCow 4
21:33 🔗 MrRadar Yeah, much better
21:34 🔗 MrRadar 22:38 < yipdw> 04:37:38 up 136 days, 8:15, 1 user, load average: 53.65, 54.40, 55.92
21:37 🔗 bwn has joined #archiveteam
21:39 🔗 SketchCow Pretty much everything is gamefront on there right now.
21:40 🔗 SketchCow After machine coughs gamefront out, it'll be better.
21:47 🔗 SketchCow In the name of making gamefront better, I shot it down to 16gb chunks instead of 40.
21:47 🔗 SketchCow More items but we can get through it
21:50 🔗 scyther has quit IRC (Leaving)
21:58 🔗 SimpBrain has joined #archiveteam
22:03 🔗 jleclanch has quit IRC (Read error: Connection reset by peer)
22:05 🔗 jleclanch has joined #archiveteam
22:06 🔗 cvb has quit IRC (Quit: Leaving)
22:09 🔗 aaaaaaaa_ has joined #archiveteam
22:09 🔗 aaaaaaaaa has quit IRC (Read error: Connection reset by peer)
22:09 🔗 swebb sets mode: +o aaaaaaaa_
22:09 🔗 aaaaaaaa_ is now known as aaaaaaaaa
22:16 🔗 Start has quit IRC (Quit: Disconnected.)
22:35 🔗 marvinw has joined #archiveteam
22:36 🔗 xmc bzc6p: i had some munin weirdness in that and other monitors. i'm not sure what changed, i upgraded one machine and several seemingly unrelated things stopped working. i'm a pretty shite sysadmin tbh.
22:41 🔗 insane_al has quit IRC (Read error: Operation timed out)
22:53 🔗 zenguy_pc has quit IRC (Ping timeout: 310 seconds)
22:55 🔗 arkiver SketchCow: meeting was great yesterday. We should do more Archive Team meetings like that
22:55 🔗 edsu has quit IRC (Ping timeout: 258 seconds)
23:01 🔗 SketchCow So expensive to just fly to Amsterdam though
23:01 🔗 edsu has joined #archiveteam
23:01 🔗 swebb sets mode: +o edsu
23:02 🔗 ersi Heh, literally 50 bucks for some of us in here tho :)
23:03 🔗 BlueMaxim has joined #archiveteam
23:05 🔗 zenguy_pc has joined #archiveteam
23:05 🔗 signius has quit IRC (Ping timeout: 310 seconds)
23:12 🔗 arkiver SketchCow: Costs are a problem, yeah
23:13 🔗 arkiver yipdw: SketchCow: if the loads are back to normal can the number of max rsync connections please be put back at 150, or at least higher then 25?
23:20 🔗 MrCave has joined #archiveteam
23:21 🔗 Start has joined #archiveteam
23:25 🔗 SketchCow No, because the machine is still suffering
23:25 🔗 SketchCow 94% disk usage, 90% of which is Gamefront
23:27 🔗 SketchCow Where's the person complaining we didn't get back about the software upload
23:32 🔗 MrCave has quit IRC (Ping timeout: 240 seconds)
23:37 🔗 Ghost_of_ has quit IRC (Quit: Leaving)
23:49 🔗 arkiver SketchCow: I can pause the grabs going to FOS if that helps
23:56 🔗 SketchCow Not clear
23:57 🔗 SketchCow Do we think that Gamefront is closing very soon?
23:58 🔗 MrRadar I don't think so, I think we were grabbing it because it seemed to be abandoned
23:58 🔗 MrRadar And they automatically delete files after 1 year
23:59 🔗 arkiver They aren't closing. They just have a horrible policy of removing files when they're not downloaded for some time.
23:59 🔗 SketchCow Right now, there are 418 15gb items sitting on fos
23:59 🔗 arkiver And they have a lot of rare/important files which aren't downloaded very often
23:59 🔗 bai so just download them and throw them away :D
23:59 🔗 bai good for another year!
23:59 🔗 SketchCow I am, as fast as the system allows, WARCing and uploading
23:59 🔗 SketchCow I suppose I could fling the directory into the second drive and see if I can run two at once.
irclogger-viewer