[00:00] *** Meeh has joined #archiveteam [00:11] *** ersi has quit IRC (Read error: Operation timed out) [00:11] *** ersi_ has joined #archiveteam [00:18] *** antomatic has quit IRC (Read error: Operation timed out) [00:27] *** Meeh has quit IRC (Read error: Connection reset by peer) [00:28] *** Meeh has joined #archiveteam [00:40] *** antomatic has joined #archiveteam [00:40] *** swebb sets mode: +o antomatic [00:46] *** xk_id has quit IRC (Remote host closed the connection) [01:07] *** mrcave_ has joined #archiveteam [01:14] *** mrcave_ has quit IRC (Quit: Page closed) [01:17] *** Ravenloft has joined #archiveteam [01:28] *** MMovie1 has quit IRC (Ping timeout: 310 seconds) [01:36] *** MMovie has joined #archiveteam [01:37] *** zenguy_pc has quit IRC (Read error: Operation timed out) [01:38] *** primus104 has quit IRC (Leaving.) [01:38] *** Coderjoe has quit IRC (Ping timeout: 255 seconds) [01:42] *** nertzy has joined #archiveteam [01:46] *** Coderjoe has joined #archiveteam [01:51] *** zenguy_pc has joined #archiveteam [02:17] *** xk_id has joined #archiveteam [02:28] *** xk_id has quit IRC (Ping timeout: 615 seconds) [02:29] Do we have all of the projects that are reliant on FOS paused right now? rsync is failing for wikis and gamefront for me. [02:37] *** JesseW has quit IRC (Leaving.) [02:53] *** RichardG has quit IRC (Read error: Connection reset by peer) [02:58] *** RichardG has joined #archiveteam [03:08] *** JesseW has joined #archiveteam [03:18] *** xk_id has joined #archiveteam [03:26] *** Start has quit IRC (Quit: Disconnected.) [03:29] *** Start has joined #archiveteam [03:33] *** xk_id has quit IRC (Ping timeout: 615 seconds) [03:38] *** zenguy_pc has quit IRC (Read error: Operation timed out) [03:52] *** zenguy_pc has joined #archiveteam [03:53] *** Ymgve has quit IRC (Ping timeout: 506 seconds) [03:56] *** Meeh has quit IRC (Ping timeout: 506 seconds) [03:58] *** Meeh has joined #archiveteam [04:00] *** Ymgve has joined #archiveteam [04:08] They seem to succeed now and again [04:12] Does it delete stuff as it finishes uploading to IA? If so that could be why [04:17] MrRadar: If I'm not mistaken, we have a handful of projects that are rather large that are consuming a good chunk of the space on FOS right now [04:18] thingiverse and blingee takes up like 7TB as is [04:18] Gamefront is 10TB [04:21] I have no idea how big FOS is in terms of disk space, but I thought there was some megawarc process that happened on FOS before it got pushed to IA [04:23] *** aaaaaaaaa has quit IRC (Leaving) [04:24] *** xk_id has joined #archiveteam [04:36] *** xk_id has quit IRC (Ping timeout: 614 seconds) [04:37] it's not a space problem [04:38] $ w [04:38] 04:37:38 up 136 days, 8:15, 1 user, load average: 53.65, 54.40, 55.92 [04:38] there's ~5 TB free on data target partitions [04:39] I don't know what error you're seeing, but if it's max connections reached or something related to load that's why [04:39] Wow, impressive load average [04:39] the remedy is forcing people to back off [04:39] which I will do now [04:39] something tragedy of the commons something [04:42] there's also a lot of stuck rsync process messages in dmesg blocked on what looks like I/O https://gist.github.com/yipdw/a60141933eb741d5014d [04:43] at the moment, there are two megawarc packers active; each packer is very I/O intensive [04:53] yipdw: Ah, ok. Out of curiosity, how much space does FOS have in total? [04:54] 16.2 TB [04:55] I don't know how much of that is available for this stuff [04:55] Gotcha. [04:56] my rsync jobs are running again [04:56] just so you know [04:56] you may see increased rates of max connections reached errors [04:56] there may be (probably is) someone running a ludicrous number of works [04:56] workers [04:57] Are you able to log by IP or something? [04:57] the tracker does yeah [04:57] Rather, do those logs exist? [04:57] also by username [04:57] ok cool [04:57] yipdw: I have 200 instances on wikis, then 200 on gamefront... [04:58] well there we go [04:58] Are we still waiting on any of the data to come back from the telenor grab? It could be HCross with the scaleway servers he was running [05:00] let me know if you want me to change my setup [05:00] I have 20 servers, each with its own ip, running 20 instanses each [05:00] Atluxity: you're running the wikiteam pipeline right? [05:00] it may be possible to adjust fos' I/O scheduling to better handle this sort of load, but I don't want to do that because fos is (supposedly) under FAI control [05:00] phuzion: yes [05:01] and I don't know anything about the FAI loadout [05:01] Atluxity: do you get lots of 410 responses for Wikia sites? [05:01] FAI? [05:01] Fully Automated Installation [05:01] I guess I could investigate runtime tweaks, the Linux I/O scheduler can be tuned that way [05:01] phuzion: jupp [05:01] Atluxity: ok, just wanted to make sure I wasn't going crazy [05:02] or getting my IP banned or something [05:02] yipdw: oh, it's config management, right? [05:02] yeah [05:02] ok [05:02] oh huh, sar is also installed, neat [05:03] huh 1200 transfers/second, I don't know where that is on the what-the-hardware-can-do scale [05:04] yipdw: Process RsyncUpload returned exit code 5 for Item file:2264798 [05:04] Is that "too many connections to the box" or what? [05:04] there's an error message that states "max connections reached" [05:04] it's printed out [05:04] exit code 5 from rsync means "error starting client-server protocol" [05:04] God damn [05:04] I really ought to read the entire output for once [05:04] lol [05:05] I do recommend people lower their worker count, fos is a bottleneck [05:05] and it's at max connections 50 anyway [05:05] ok [05:05] I recognize myself as people [05:05] to scale higher we need more rsync targets [05:06] yipdw: FOS runs debian, right? Jessie? [05:07] Ubuntu [05:07] or at least that's what /etc/issue says [05:07] it has apt so I guess that's close enough [05:07] Ah, ok. I'm gonna see if there's any optimizations that can be made on the rsync config. Do you know if rsync is using the distro default config? [05:08] *** bzc6p_ has joined #archiveteam [05:08] *** swebb sets mode: +o bzc6p_ [05:08] er wait [05:08] argh that /etc/issue was my own system [05:08] where the hell is my fos terminal [05:08] I'll stop my workers, then set them up with new instance count [05:08] well it's still Ubuntu heh [05:09] phuzion: yes, aside from module configurations, I don't see anything custom in rsyncd.conf [05:09] *** bzc6p has quit IRC (Read error: Operation timed out) [05:09] *** PepsiMax_ has joined #archiveteam [05:09] any idea what will be a more sane instance count? [05:09] ok, I'll see if there's anything I can do to optimize rsyncd to accept more connections to reduce this bottleneck. [05:09] well, we can accept more connections fine [05:10] the I/O subsystem can't keep up [05:10] Ohhh [05:10] Atluxity: I'm not sure, perhaps start low and scale slowly [05:10] I thought rsync was rejecting the connection because it couldn't open a connection for some reason [05:10] *** dcmorton has quit IRC (Read error: Operation timed out) [05:11] *** dserodio has quit IRC (Read error: Operation timed out) [05:11] *** HarryCros has quit IRC (Read error: Operation timed out) [05:11] yeah, if you saw max connections reached, it's because the max connection count for that module was hit [05:11] *** Sanqui has quit IRC (Ping timeout: 499 seconds) [05:11] but I set that explicitly because at the old limit fos was getting killed [05:11] Ohhhhhhh, so the limit is placed in rsyncd artificially so we don't murder IO ok [05:11] old limit was 150; we're at 25 now (50 still didn't seem to help) [05:12] *** robink has quit IRC (Ping timeout: 499 seconds) [05:13] I assume the IO isn't due to sheer write speed, it's the insane number of transactions we're putting on the disk, right? [05:13] I don't know [05:13] I do know that there are two gamefront megawarc packers running [05:13] gamefront is videos, right [05:13] I'm not sure. [05:14] I've seen items anywhere between 0.1MB to 3-500MB [05:14] 562MB just hit on the tracker [05:15] yipdw: If you had to guess, how long would those megawarc packers take to run? [05:15] *** PepsiMax has quit IRC (Ping timeout: 730 seconds) [05:15] Also, how much data is being packed? [05:15] *** chazchaz has quit IRC (Ping timeout: 369 seconds) [05:16] don't know [05:16] *** atlogbot has quit IRC (Ping timeout: 369 seconds) [05:16] I'm wondering if putting a couple of 512GB SSDs in FOS would help for megawarc packing [05:16] probably [05:16] maintenance cost goes up though [05:17] I'll ask slappfisk to back down a bit too... he also has access to many servers [05:17] I'd bet we could crowdfund $500 for 2x512GB SSDs pretty easily. I'd probably chip in $50 or something [05:18] If that would be anywhere close to enough storage for megawarc packing, that is. [05:18] it's not the initial thing that worries me, it's upkeep [05:18] SSDs are gonna die [05:19] True. [05:19] and all that upkeep goes into IA's budget [05:19] Right. Maybe some 15K SAS drives then :) [05:19] unless we continue that funding thing as a recurring donation or something [05:19] Less likely to be murdered by IO over time. [05:20] But also significantly more expensive. [05:21] *** kniffy has quit IRC (Ping timeout: 369 seconds) [05:22] yipdw: Perhaps a stupid question, but how is the storage on FOS split up? One big mount or multiple smaller mounts? [05:23] two RAID devices, each with their own mount [05:23] Are megawarcs packed on one and incoming files for rsync on another? [05:24] Or is it just "here's bin A, here's bin B, do whatever on either" [05:25] megawarc chunking and packing need to be done on the same filesystem, packing -> uploading can move elsewhere [05:25] or rather https://github.com/ArchiveTeam/archiveteam-megawarc-factory#filesystems explains it better [05:26] *** kniffy has joined #archiveteam [05:26] Ok, that makes sense. [05:28] I'm not sure which filesystem the packer is on [05:28] yet [05:29] I think it's on filesystem 0, all rsync goes to filesystem 1 [05:29] and I'm not sure the chunker is running [05:29] it doesn't look like it [05:29] oh wait it's not the same chunker I know [05:29] pgrep chunker doesn't return anything on the system? [05:29] interesting [05:29] oh [05:30] I am uh not going to touch that [05:32] *** kniffy has quit IRC (Quit: :^)) [05:33] *** kniffy has joined #archiveteam [05:37] yipdw: heyo, have some stuff to ask about ArchiveBot, mind if I PM you? [05:39] *** atlogbot has joined #archiveteam [05:39] *** HarryCros has joined #archiveteam [05:40] *** chazchaz has joined #archiveteam [05:40] *** zenguy_pc has quit IRC (Read error: Operation timed out) [05:41] *** dserodio has joined #archiveteam [05:41] *** dcmorton has joined #archiveteam [05:42] *** kniffy has quit IRC (Quit: :^)) [05:43] *** kniffy has joined #archiveteam [05:43] *** kniffy has quit IRC (Remote host closed the connection) [05:43] *** kniffy has joined #archiveteam [05:53] *** zenguy_pc has joined #archiveteam [05:53] *** xk_id has joined #archiveteam [05:54] *** dcmorton has quit IRC (Read error: Operation timed out) [05:54] *** dserodio has quit IRC (Read error: Operation timed out) [05:55] *** atlogbot has quit IRC (Read error: Operation timed out) [05:56] *** HarryCros has quit IRC (Read error: Operation timed out) [05:57] *** WinterFox has joined #archiveteam [05:57] *** kniffy has quit IRC (Quit: :^)) [05:57] *** kniffy has joined #archiveteam [06:00] *** chazchaz has quit IRC (Ping timeout: 369 seconds) [06:04] *** Sanqui has joined #archiveteam [06:04] *** slyphic has quit IRC (Read error: Operation timed out) [06:05] *** Deewiant has joined #archiveteam [06:06] *** HarryCros has joined #archiveteam [06:06] *** atlogbot has joined #archiveteam [06:06] *** chazchaz has joined #archiveteam [06:07] *** slyphic has joined #archiveteam [06:08] dan-: yeah sure that's cool [06:08] *** dserodio has joined #archiveteam [06:08] *** dcmorton has joined #archiveteam [06:25] *** cvb has joined #archiveteam [06:33] *** Sanqui has quit IRC (Ping timeout: 499 seconds) [06:37] *** cvb has quit IRC (Read error: Operation timed out) [06:38] *** cvb has joined #archiveteam [06:38] *** HarryCros has quit IRC (Read error: Operation timed out) [06:41] *** chazchaz has quit IRC (Read error: Operation timed out) [06:41] *** atlogbot has quit IRC (Ping timeout: 369 seconds) [06:43] *** dserodio has quit IRC (Read error: Operation timed out) [06:44] *** slyphic has quit IRC (Read error: Operation timed out) [06:49] *** primus104 has joined #archiveteam [06:54] *** xk_id has quit IRC (Remote host closed the connection) [06:54] *** chazchaz has joined #archiveteam [06:55] *** dcmorton has quit IRC (Ping timeout: 369 seconds) [06:55] there... backing down from 2 x 10 x 20 to 2 x 10 x 5 seems to have helped the flow of items quite a bit [07:00] Atluxity: feel free to toss the others at urlteam, I think. :-) [07:01] *** cvb has quit IRC (Quit: Leaving) [07:02] JesseW: I'm fairly certain URLteam has enough workers on it, but more certainly can't hurt in the long run. [07:02] JesseW: got 200 instances at urlteam already :) [07:02] Jesus [07:03] but I did not notice that it increased scans per second, so I dont know what the bottleneck is [07:03] Atluxity: Who do you host all this with? [07:03] phuzion: I am testing a new openstack rig at work [07:04] Ahhh, jealous [07:04] oh jesus [07:04] I could do whatever, just as long as I kept under 1Gbit\s and did not expect anything to stay alive or be kept safe [07:05] Haha christ, that's insane. "Here's some crazy new gear to mess around with. Keep the network traffic under 1gbps. Have fun." [07:05] so I dont think this will be a nice place for archivebot instance :\ [07:06] :( [07:06] *** dserodio has joined #archiveteam [07:06] *** dcmorton has joined #archiveteam [07:07] *** atlogbot has joined #archiveteam [07:08] *** HarryCros has joined #archiveteam [07:08] yipdw: Mind if I pm you? I had a couple questions if you have time [07:09] *** slyphic has joined #archiveteam [07:09] Jordan_: sure, that's cool [07:09] *** Kenshin has quit IRC (Read error: Operation timed out) [07:10] phuzion: my deploy-script is a titanpad piped to shell as root.... *giggles* [07:10] you monster [07:10] lol [07:10] fun to have servers one does not care about [07:13] holy crap, gamefront got 59K items out [07:13] *** Kenshin has joined #archiveteam [07:14] Yeah, we might wanna see about getting some of the older items re-queued [07:17] who are these random new (at least, new to me) people with private questions for yipdw ...? [07:19] they seems to scare to ask directly? [07:20] Jordan: randomly several people have asked yipdw about privmsg for questions in a short time. Is something going on? [07:20] Jordan_ even [07:20] https://www.youtube.com/watch?v=xxMb_8ZpjIY [07:21] *** slyphic has quit IRC (Read error: Operation timed out) [07:21] *** primus104 has quit IRC (Leaving.) [07:21] must be it. :-) [07:21] they're ArchiveBot-related questions [07:22] *** dserodio has quit IRC (Read error: Operation timed out) [07:23] *** dcmorton has quit IRC (Read error: Operation timed out) [07:23] just seems unusual to have two people ask to ask in quick succession [07:23] but, not my business, so [07:24] *** HarryCros has quit IRC (Read error: Operation timed out) [07:27] yipdw: is there a way to view the queue on archivebot? [07:27] !pending [07:27] Web interface? [07:28] oh, no [07:28] ok [07:28] was just curious [07:34] Jeez, there's jobs running since June?!? [07:35] *** atlogbot has quit IRC (Ping timeout: 369 seconds) [07:37] *** zenguy_pc has quit IRC (Read error: Operation timed out) [07:38] Question about the Gamefront grab. Is it normal to have dozens or hundreds of hits to facebook when running an item for that job? [07:41] *** atlogbot has joined #archiveteam [07:41] *** jleclanch has quit IRC (Read error: Connection reset by peer) [07:42] *** slyphic has joined #archiveteam [07:42] *** HarryCros has joined #archiveteam [07:42] *** jleclanch has joined #archiveteam [07:43] *** dcmorton has joined #archiveteam [07:44] *** dserodio has joined #archiveteam [07:47] I hope so [07:47] I see alot of that [07:51] If it isn't, we could probably speed up the retrieval of items by blocking those. [07:53] *** zenguy_pc has joined #archiveteam [07:55] *** jleclanch has quit IRC (Read error: Connection reset by peer) [07:57] *** jleclanch has joined #archiveteam [08:01] yeah [08:07] *** jleclanch has quit IRC (Read error: Connection reset by peer) [08:09] *** jleclanch has joined #archiveteam [08:11] *** JesseW has quit IRC (Leaving.) [08:16] *** jleclanch has quit IRC (Read error: Connection reset by peer) [08:18] *** jleclanch has joined #archiveteam [08:18] *** jleclanch has quit IRC (Remote host closed the connection) [08:25] *** xk_id has joined #archiveteam [08:30] *** jleclanch has joined #archiveteam [08:34] *** xk_id has quit IRC (Read error: Operation timed out) [08:36] *** jleclanch has quit IRC (Read error: Operation timed out) [08:39] *** jleclanch has joined #archiveteam [08:50] *** jleclanch has quit IRC (Remote host closed the connection) [08:55] *** primus104 has joined #archiveteam [09:05] *** ersi_ is now known as ersi [09:11] *** atomotic has joined #archiveteam [09:31] *** Stiletto has quit IRC (Ping timeout: 272 seconds) [09:32] *** Stiletto has joined #archiveteam [09:34] *** schbirid has joined #archiveteam [09:41] *** zenguy_pc has quit IRC (Read error: Operation timed out) [09:49] *** scyther has joined #archiveteam [09:50] *** Guest7171 has joined #archiveteam [09:51] yipdw: FOS was running for weeks with 1 or 2 megawarc packers running and uploading and 150 rsync connections. Now it suddenly can't handle it anymore? [09:52] The issue in the wikis grab with status code 410 was fixed a few days ago, I forgot to set the new version in the tracker. Will do that now [09:52] Yesterday actually ^ [09:53] *** zenguy_pc has joined #archiveteam [10:03] *** primus104 has quit IRC (Leaving.) [10:06] sigh [10:08] starting stopping this train is not as easy as I would like... but... :P [10:08] oh wait... [10:08] arkiver: so if I have pulled the git repo last 24 hours then I have the new version and dont need to do anything? [10:09] If you don't see any message that you should update your scripts you're fine [10:10] Are you able to bring all the concurrent you had online back up if needed? [10:10] (not now yet) [10:10] yeah [10:10] ok, awesome [10:10] I think they could do more concurrent too, the load was not high [10:14] sounds good [10:15] when we figure out what's going on with FOS or have a new rsync target we might need them :) [10:19] *** midas1 is now known as midas [10:27] *** xk_id has joined #archiveteam [10:37] going to test to stop all my work to see what the impact is in general [10:37] spent some effort making it easier to stop and start stuff [10:39] *** xk_id has quit IRC (Ping timeout: 615 seconds) [10:42] *** jleclanch has joined #archiveteam [10:46] *** scyther_ has joined #archiveteam [10:53] *** scyther has quit IRC (Read error: Operation timed out) [10:54] *** Muad-Dib has joined #archiveteam [11:20] *** Guest7171 has quit IRC (Quit: Textual IRC Client: www.textualapp.com) [11:29] *** xk_id has joined #archiveteam [11:37] *** xk_id has quit IRC (Read error: Operation timed out) [11:40] *** is- has quit IRC (Read error: Operation timed out) [11:41] *** is- has joined #archiveteam [11:42] *** zenguy_pc has quit IRC (Read error: Operation timed out) [11:54] *** zenguy_pc has joined #archiveteam [11:58] *** vitzli has joined #archiveteam [12:11] *** WinterFox has quit IRC (Remote host closed the connection) [12:28] *** atomotic has quit IRC (Quit: Textual IRC Client: www.textualapp.com) [12:30] *** xk_id has joined #archiveteam [12:41] *** xk_id has quit IRC (Ping timeout: 615 seconds) [12:57] *** BlueMaxim has quit IRC (Quit: Leaving) [13:03] *** rduser has quit IRC (Read error: Operation timed out) [13:03] *** scyther_ has quit IRC (Ping timeout: 272 seconds) [13:17] I do have a FOS suggestion. EU FOS? [13:20] *** arkiver2 has joined #archiveteam [13:23] *** khaoohs has quit IRC (Read error: Operation timed out) [13:31] *** xk_id has joined #archiveteam [13:31] Uh, no? [13:38] arkiver: thanks for fixing that 410 error :) [13:41] I've got 20 threads on the wiki job now [13:42] *** xk_id has quit IRC (Ping timeout: 615 seconds) [13:47] ersi: why? [13:49] *** jspiros has joined #archiveteam [13:49] Isn't the sole point of FOS that it's in IA's network? and that it got storage to buffer for a while? [13:53] ersi: the idea of FOS is that it's a staging server that can take in a ton of data while it's being collected by the various scripts/warriors. Its location doesn't really matter, as long as there's suitable bandwidth between FOS and IA. The fact that it's on IA's network is convenient though, because it means that there's no need to pay for bandwidth twice to transfer said data. [13:54] Right now, FOS's bottleneck isn't based on geographic constraints, it's I/O bound at the moment. [13:54] How redundant. Thanks for basically saying the same thing, but phrasing it differently. :) [13:54] So, I'm not sure what good a FOS in EU would serve? [13:56] * ersi nods agreeingly [13:57] biggest issue is getting stuff from fos in IA anyway [13:57] or from any host to IA for that matter :p [13:57] *** arkiver2 has quit IRC (Ping timeout: 252 seconds) [14:01] *** Sanqui has joined #archiveteam [14:08] *** arkiver2 has joined #archiveteam [14:21] *** primus104 has joined #archiveteam [14:21] *** lytv has quit IRC (Read error: Operation timed out) [14:22] *** lytv has joined #archiveteam [14:30] *** arkiver2 has quit IRC (Ping timeout: 252 seconds) [14:33] *** xk_id has joined #archiveteam [14:36] *** nertzy has quit IRC (Quit: This computer has gone to sleep) [14:41] *** xk_id has quit IRC (Read error: Connection reset by peer) [14:41] *** xk_id has joined #archiveteam [14:52] *** xk_id has quit IRC (Ping timeout: 615 seconds) [15:02] *** arkiver2 has joined #archiveteam [15:11] *** xk_id has joined #archiveteam [15:13] *** arkiver2 has quit IRC (Ping timeout: 252 seconds) [15:17] *** rduser has joined #archiveteam [15:17] *** Start has quit IRC (Quit: Disconnected.) [15:26] *** atomotic has joined #archiveteam [15:38] *** cvb has joined #archiveteam [15:42] *** Start has joined #archiveteam [15:42] *** zenguy_pc has quit IRC (Read error: Operation timed out) [15:48] *** atomotic has quit IRC (Quit: Textual IRC Client: www.textualapp.com) [15:54] *** vitzli has quit IRC (Quit: Leaving) [15:55] *** zenguy_pc has joined #archiveteam [15:55] maybe IA could start a europe-branch? [15:55] then it could have a europe-FOS [16:04] *** zenguy_pc has quit IRC (Ping timeout: 310 seconds) [16:12] *** primus104 has quit IRC (Leaving.) [16:16] *** zenguy_pc has joined #archiveteam [16:21] *** Ghost_of_ has joined #archiveteam [16:24] Atluxity: I'm not quite sure I'm following the need for a european FOS. Are we seeing any kind of performance issues that would be resolved by having a staging target geographically closer to some of the script runners? [16:29] *** ohhdemgir has joined #archiveteam [16:31] *** MMovie has quit IRC (Ping timeout: 310 seconds) [16:34] Atluxity: I don't believe the bottleneck exists on a network level [16:35] its an IO issue [16:41] *** MMovie has joined #archiveteam [16:42] *** HCross has left Leaving [16:44] arkiver: I just report what I see, you can take it up with w and sar [16:45] Legit question, what's the bare minimum size we could get away with using as a separate rsync target? Say smaller jobs. 2TB? 5? 10? [16:45] *** MMovie1 has joined #archiveteam [16:47] Depends on the project.. [16:48] *** MMovie has quit IRC (Ping timeout: 310 seconds) [16:50] Obviously a 2TB FOS wouldn't be any good for Gamefront, but say telenor? That was 237GB. I'm wondering if it would be worth it to grab a 2TB Kimsufi server or something and offer it as a secondary staging server [17:01] *** JesseW has joined #archiveteam [17:02] *** atomotic has joined #archiveteam [17:02] *** marvinw_ has quit IRC (Read error: Operation timed out) [17:04] *** i0npulse has quit IRC (Ping timeout: 506 seconds) [17:10] *** Start has quit IRC (Read error: Operation timed out) [17:11] *** scyther has joined #archiveteam [17:12] *** Start has joined #archiveteam [17:15] *** JesseW has quit IRC (Leaving.) [17:16] *** bzc6p__ has joined #archiveteam [17:16] *** swebb sets mode: +o bzc6p__ [17:18] *** bzc6p_ has quit IRC (Read error: Operation timed out) [17:34] *** primus104 has joined #archiveteam [17:39] *** arkiver2 has joined #archiveteam [17:48] *** arkiver2 has quit IRC (Ping timeout: 252 seconds) [17:51] *** riz has quit IRC (Remote host closed the connection) [17:52] *** riz has joined #archiveteam [18:04] *** zenguy_pc has quit IRC (Ping timeout: 310 seconds) [18:09] phuzion: mind that kimsufi's are single disk [18:09] phuzion: has a tendency of murdering your I/O [18:09] for projects with small files [18:09] because too many random writes [18:09] phuzion: only if you're willing to keep it online [18:10] IME the problem is rarely related to technical specifications [18:12] like, the really cool thing about fos is that it's been around long enough that we can be reasonably sure it's going to be around. this goes also for the amazing stuff Kenshin has thrown in [18:12] one-off rsyncs are fine, though they can be a bit of a pain to wrangle [18:16] *** zenguy_pc has joined #archiveteam [18:34] *** jspiros has quit IRC (Ping timeout: 186 seconds) [18:36] *** Start has quit IRC (Quit: Disconnected.) [18:42] *** joepie91 has quit IRC (Read error: Operation timed out) [18:44] *** Ghost_of_ has quit IRC (Quit: Leaving) [18:44] *** joepie91 has joined #archiveteam [18:48] *** aaaaaaaaa has joined #archiveteam [18:48] *** swebb sets mode: +o aaaaaaaaa [18:51] *** aaaaaaaaa has quit IRC (Client Quit) [18:57] *** atomotic_ has joined #archiveteam [18:59] *** atomotic has quit IRC (Ping timeout: 252 seconds) [19:02] *** jspiros has joined #archiveteam [19:11] *** Start has joined #archiveteam [19:16] *** nertzy has joined #archiveteam [19:19] *** Start has quit IRC (Quit: Disconnected.) [19:35] *** nertzy has quit IRC (Quit: This computer has gone to sleep) [19:41] *** vOYtEC has quit IRC (Read error: Connection reset by peer) [19:48] *** insane_al has joined #archiveteam [19:52] *** SimpBrain has quit IRC (Leaving) [19:54] *** aaaaaaaaa has joined #archiveteam [19:54] *** swebb sets mode: +o aaaaaaaaa [20:05] *** zenguy_pc has quit IRC (Ping timeout: 310 seconds) [20:17] *** zenguy_pc has joined #archiveteam [20:33] *** aaaaaaaa_ has joined #archiveteam [20:33] *** aaaaaaaaa has quit IRC (Read error: Connection reset by peer) [20:33] *** swebb sets mode: +o aaaaaaaa_ [20:34] *** aaaaaaaa_ is now known as aaaaaaaaa [20:35] Sold out largest Archive Team meeting for a BURGER [20:41] *** Ghost_of_ has joined #archiveteam [20:43] *** bzc6p__ is now known as bzc6p [20:45] FOR. A. BURGER. [20:45] *** aaaaaaaaa has quit IRC (Read error: Connection reset by peer) [20:45] Wow, people had lots of FOS talk while I'm gone. [20:46] *** aaaaaaaaa has joined #archiveteam [20:46] *** swebb sets mode: +o aaaaaaaaa [20:46] indeed :) [20:46] SketchCow: Yeah, I'm not sure how I can be helpful short of running scripts, so I'm trying to think of solutions to problems as I see them :) [20:47] *** aaaaaaaaa has quit IRC (Client Quit) [20:47] some ramlbing about euro-FOS, think everyone agreed that can be ignored [20:47] A lot of it was "I'm not on the machine, I don't know the stats, I can't really come up with coherent solutions without that information, but WHY LET THAT STOP ME NOW" <30 lines> [20:48] cause solving problems are fun! who cares if they are there or not :D [20:48] lol [20:49] Things worth noting: [20:49] 1. FOS is a VM, not its own server [20:49] 2. Some things, the scripts are running and do the work. Some I do manually. [20:49] 3. I was in the country of Europe all last week [20:49] 4. Since as we know, the vast social net built into their pinko government systems means free blowjobs forever, I was busy [20:50] * ersi shrugs [20:50] 5. I didn't know about the disk full until Arkiver told me, and I've been fighting that back [20:50] Country of Europe [20:50] * ersi snickers [20:50] 6. I suspect that there have been some notable jobs added which have no "pack and put into IA" process attached yet., [20:52] *** zenguy_pc has quit IRC (Ping timeout: 310 seconds) [20:52] So I'm quickly giving FOS the once over. I also restarted rsync in case you maniancs broke its heart or something. [20:53] I might have been a bit eager [20:55] *** Start has joined #archiveteam [20:55] SketchCow: where was this mythical burger bar that you spoke of [20:55] Out the theater door, to the left, 5 doors down. [20:56] *** atomotic_ has quit IRC (Quit: Textual IRC Client: www.textualapp.com) [20:56] SketchCow: that's the direction in which we walked, but we failed to observe a burger bar... [20:56] then again, we had to catch a train anyway :P [20:56] It was RIGHT there [20:56] Schoutenstraat 15, 3512 GA Utrecht [20:57] The nice Brazilian Hagrid found us [20:57] Wrong address I gave you. [20:58] Anyway, it was nice. [20:58] Then I went to bed, got up nice and early, and then 9 hour flight to here. [20:59] SketchCow: is it bad that I immediately know who you are refering to with "Brazilian Hagrid" [20:59] lol [21:04] *** zenguy_pc has joined #archiveteam [21:05] FYI: Getting strange rsync errors randomly on gamefront, http://pastebin.com/iGxWUiq9 [21:06] phuzion: Regarding ArchiveBot queue web interface, we have this, quite nice: http://archivebot.at.ninjawedding.org:4567/ [21:07] matthusby: not seen that variant before [21:08] today, it seems to me that the pipelines are set up to grab one item, then rsync it to some stageing place (FOS) [21:08] xmc: I miss the fancy graphs of projects' performance on http://zeppelin.xrtc.net. Not updated since July 28. If it just needs a restart, it would be nice to have it. Thanks in advance. [21:09] it seems to me like an rsync session is better for large transfers, and that small ones are not optimal, could perhaps the pipeline rsync after 5 items or after X items instead? would that solve anything? [21:10] Atluxity: the limit on simultaneous rsync connections is artificially imposed to keep the I/O from being killed on FOS [21:11] ah, ok [21:12] Check logs starting about 23:15 Eastern last night, we discussed the rsync limits and whatnot [21:12] *** schbirid has quit IRC (Quit: Leaving) [21:17] *** aaaaaaaaa has joined #archiveteam [21:17] *** swebb sets mode: +o aaaaaaaaa [21:22] g'nite [21:25] SOMETHING got better on IO [21:25] Because FOS seems (seems) zippy again [21:31] What's the load average? it was up in the 50s last night [21:33] 4 [21:33] Yeah, much better [21:34] 22:38 < yipdw> 04:37:38 up 136 days, 8:15, 1 user, load average: 53.65, 54.40, 55.92 [21:37] *** bwn has joined #archiveteam [21:39] Pretty much everything is gamefront on there right now. [21:40] After machine coughs gamefront out, it'll be better. [21:47] In the name of making gamefront better, I shot it down to 16gb chunks instead of 40. [21:47] More items but we can get through it [21:50] *** scyther has quit IRC (Leaving) [21:58] *** SimpBrain has joined #archiveteam [22:03] *** jleclanch has quit IRC (Read error: Connection reset by peer) [22:05] *** jleclanch has joined #archiveteam [22:06] *** cvb has quit IRC (Quit: Leaving) [22:09] *** aaaaaaaa_ has joined #archiveteam [22:09] *** aaaaaaaaa has quit IRC (Read error: Connection reset by peer) [22:09] *** swebb sets mode: +o aaaaaaaa_ [22:09] *** aaaaaaaa_ is now known as aaaaaaaaa [22:16] *** Start has quit IRC (Quit: Disconnected.) [22:35] *** marvinw has joined #archiveteam [22:36] bzc6p: i had some munin weirdness in that and other monitors. i'm not sure what changed, i upgraded one machine and several seemingly unrelated things stopped working. i'm a pretty shite sysadmin tbh. [22:41] *** insane_al has quit IRC (Read error: Operation timed out) [22:53] *** zenguy_pc has quit IRC (Ping timeout: 310 seconds) [22:55] SketchCow: meeting was great yesterday. We should do more Archive Team meetings like that [22:55] *** edsu has quit IRC (Ping timeout: 258 seconds) [23:01] So expensive to just fly to Amsterdam though [23:01] *** edsu has joined #archiveteam [23:01] *** swebb sets mode: +o edsu [23:02] Heh, literally 50 bucks for some of us in here tho :) [23:03] *** BlueMaxim has joined #archiveteam [23:05] *** zenguy_pc has joined #archiveteam [23:05] *** signius has quit IRC (Ping timeout: 310 seconds) [23:12] SketchCow: Costs are a problem, yeah [23:13] yipdw: SketchCow: if the loads are back to normal can the number of max rsync connections please be put back at 150, or at least higher then 25? [23:20] *** MrCave has joined #archiveteam [23:21] *** Start has joined #archiveteam [23:25] No, because the machine is still suffering [23:25] 94% disk usage, 90% of which is Gamefront [23:27] Where's the person complaining we didn't get back about the software upload [23:32] *** MrCave has quit IRC (Ping timeout: 240 seconds) [23:37] *** Ghost_of_ has quit IRC (Quit: Leaving) [23:49] SketchCow: I can pause the grabs going to FOS if that helps [23:56] Not clear [23:57] Do we think that Gamefront is closing very soon? [23:58] I don't think so, I think we were grabbing it because it seemed to be abandoned [23:58] And they automatically delete files after 1 year [23:59] They aren't closing. They just have a horrible policy of removing files when they're not downloaded for some time. [23:59] Right now, there are 418 15gb items sitting on fos [23:59] And they have a lot of rare/important files which aren't downloaded very often [23:59] so just download them and throw them away :D [23:59] good for another year! [23:59] I am, as fast as the system allows, WARCing and uploading [23:59] I suppose I could fling the directory into the second drive and see if I can run two at once.