[00:00] *** mafrasi2_ has quit IRC (No Ping reply in 180 seconds.) [00:03] *** mafrasi2 has joined #archiveteam [00:04] *** mhazinsk has joined #archiveteam [00:05] *** rossdylan has joined #archiveteam [00:07] *** aMunster has joined #archiveteam [00:15] *** Famicoma1 has quit IRC (Read error: Connection reset by peer) [00:17] *** Famicoma1 has joined #archiveteam [00:17] *** nwf has joined #archiveteam [00:36] *** closure has joined #archiveteam [00:55] *** MMovie has joined #archiveteam [01:01] *** Jonimus has joined #archiveteam [01:12] *** fie has joined #archiveteam [01:12] *** JesseW has joined #archiveteam [01:15] *** fie_ has quit IRC (Ping timeout: 258 seconds) [01:16] *** pikhq has joined #archiveteam [01:19] *** JesseW has quit IRC (Quit: Leaving.) [01:29] *** Start has joined #archiveteam [02:08] *** philpem has quit IRC (Ping timeout: 260 seconds) [02:28] *** JesseW has joined #archiveteam [02:46] *** Guest__ has quit IRC (Quit: My MacBook Pro has gone to sleep. ZZZzzz…) [02:58] *** Zebranky has quit IRC (Read error: Operation timed out) [02:58] *** Zebranky has joined #archiveteam [03:22] *** Apathy has quit IRC (Read error: Operation timed out) [03:24] *** Apathy has joined #archiveteam [03:38] *** Ungstein has quit IRC (Quit: Leaving.) [03:41] *** w0rp has quit IRC (Ping timeout: 258 seconds) [03:41] *** nekomune has quit IRC (Ping timeout: 258 seconds) [03:42] *** w0rp has joined #archiveteam [03:42] *** pikhq has quit IRC (Ping timeout: 258 seconds) [03:42] *** Peetz0r has quit IRC (Ping timeout: 258 seconds) [03:42] *** Frogging has quit IRC (Ping timeout: 258 seconds) [03:42] *** Peetz0r has joined #archiveteam [03:42] *** Frogging has joined #archiveteam [03:43] *** GChriss has quit IRC (Ping timeout: 258 seconds) [03:43] *** MrRadar has quit IRC (Ping timeout: 258 seconds) [03:43] *** chfoo has quit IRC (Ping timeout: 258 seconds) [03:43] *** goekesmi has quit IRC (Ping timeout: 258 seconds) [03:43] *** espes__ has quit IRC (Ping timeout: 258 seconds) [03:43] *** goekesmi has joined #archiveteam [03:43] *** chfoo has joined #archiveteam [03:43] *** Ungstein has joined #archiveteam [03:44] *** espes__ has joined #archiveteam [03:45] *** MrRadar has joined #archiveteam [03:45] *** GChriss has joined #archiveteam [03:45] *** kcaj has quit IRC (Ping timeout: 258 seconds) [03:47] *** Lord_Nigh has quit IRC (Ping timeout: 258 seconds) [03:48] *** kcaj has joined #archiveteam [03:48] *** MrRadar_ has joined #archiveteam [03:49] *** MrRadar_ has quit IRC (Client Quit) [03:51] *** ersi has quit IRC (hub.dk irc.underworld.no) [03:51] *** z00nx has quit IRC (hub.dk irc.underworld.no) [03:51] *** will has quit IRC (hub.dk irc.underworld.no) [03:51] *** Marc has quit IRC (hub.dk irc.underworld.no) [03:51] *** useretail has quit IRC (hub.dk irc.underworld.no) [03:51] *** Rye has quit IRC (hub.dk irc.underworld.no) [03:51] *** Nemo_bis has quit IRC (hub.dk irc.underworld.no) [03:52] *** Lord_Nigh has joined #archiveteam [03:54] *** pikhq has joined #archiveteam [03:56] *** will__ has joined #archiveteam [03:57] *** ersi_ has joined #archiveteam [03:57] *** bwn has quit IRC (Read error: Operation timed out) [04:07] *** will__ is now known as will [04:32] *** wp494 has quit IRC (Read error: Connection reset by peer) [05:08] *** Sk1d has quit IRC (Ping timeout: 194 seconds) [05:14] *** Sk1d has joined #archiveteam [05:32] *** wp494 has joined #archiveteam [05:50] *** JesseW has quit IRC (Quit: Leaving.) [06:03] *** WinterFox has joined #archiveteam [06:28] *** fie_ has joined #archiveteam [06:30] *** fie has quit IRC (Read error: Operation timed out) [07:10] *** roninski has quit IRC (Read error: Connection reset by peer) [07:11] *** roninski has joined #archiveteam [07:20] *** ersi_ is now known as ersi [07:23] *** roninski has quit IRC (Read error: Connection timed out) [07:24] *** roninski has joined #archiveteam [07:32] *** RichardG_ has joined #archiveteam [07:32] *** RichardG has quit IRC (Ping timeout: 260 seconds) [07:34] *** metalcamp has joined #archiveteam [07:38] *** bwn has joined #archiveteam [07:40] *** RichardG has joined #archiveteam [07:41] *** RichardG_ has quit IRC (Ping timeout: 258 seconds) [07:41] *** useretail has joined #archiveteam [07:41] *** Nemo_bis has joined #archiveteam [07:41] *** Rye has joined #archiveteam [07:41] *** Marc has joined #archiveteam [07:43] *** z00nx has joined #archiveteam [07:45] *** nekomune has joined #archiveteam [08:00] *** RichardG_ has joined #archiveteam [08:04] *** RichardG has quit IRC (Read error: Operation timed out) [08:06] *** metal_cam has joined #archiveteam [08:08] *** metalcamp has quit IRC (Ping timeout: 250 seconds) [08:17] *** vOYtEC has quit IRC (rm -r *) [08:17] *** metal_cam is now known as metalcamp [08:19] *** ndiddy has quit IRC (Read error: Connection reset by peer) [08:23] *** Cameron_D has joined #archiveteam [08:50] *** RichardG has joined #archiveteam [08:54] *** Guest_ has joined #archiveteam [08:54] *** atomotic has joined #archiveteam [08:55] *** RichardG_ has quit IRC (Ping timeout: 633 seconds) [08:56] *** metalcamp has quit IRC (Ping timeout: 244 seconds) [09:11] *** RichardG_ has joined #archiveteam [09:16] *** RichardG has quit IRC (Read error: Operation timed out) [09:17] *** bwn has quit IRC (Ping timeout: 492 seconds) [09:33] *** RichardG has joined #archiveteam [09:34] *** brabo has quit IRC (Read error: Operation timed out) [09:35] *** RichardG_ has quit IRC (Ping timeout: 272 seconds) [09:38] *** bwn has joined #archiveteam [09:42] *** roninski1 has joined #archiveteam [09:43] *** rossdylan has quit IRC (Read error: Operation timed out) [09:44] *** roninski has quit IRC (Ping timeout: 260 seconds) [09:44] *** brabo has joined #archiveteam [09:56] *** rossdylan has joined #archiveteam [09:56] *** jut has joined #archiveteam [10:06] *** schbirid has joined #archiveteam [11:04] *** SmileyG has joined #archiveteam [11:04] *** winr5r has joined #archiveteam [11:05] *** winr4r has quit IRC (Read error: Operation timed out) [11:05] *** Jogie has quit IRC (Read error: Operation timed out) [11:06] *** Mayeau has joined #archiveteam [11:06] *** lbft_ has joined #archiveteam [11:06] *** Mayonaise has quit IRC (Read error: Operation timed out) [11:08] *** roninski has joined #archiveteam [11:08] *** roninski1 has quit IRC (Ping timeout: 260 seconds) [11:09] *** schbirid has quit IRC (hub.efnet.us irc.Prison.NET) [11:09] *** Zebranky has quit IRC (hub.efnet.us irc.Prison.NET) [11:09] *** sivoais has quit IRC (hub.efnet.us irc.Prison.NET) [11:09] *** Smiley has quit IRC (hub.efnet.us irc.Prison.NET) [11:09] *** patrickod has quit IRC (hub.efnet.us irc.Prison.NET) [11:09] *** lbft has quit IRC (hub.efnet.us irc.Prison.NET) [11:09] *** oli has quit IRC (hub.efnet.us irc.Prison.NET) [11:09] *** achip has quit IRC (hub.efnet.us irc.Prison.NET) [11:09] *** oli_ has joined #archiveteam [11:15] *** schbirid2 has joined #archiveteam [11:16] *** achip has joined #archiveteam [11:17] *** patrickod has joined #archiveteam [11:20] *** Zebranky_ has joined #archiveteam [11:24] *** oli_ is now known as oli [11:31] *** metalcamp has joined #archiveteam [11:38] *** sivoais has joined #archiveteam [11:43] *** metal_cam has joined #archiveteam [11:45] *** metalcamp has quit IRC (Ping timeout: 244 seconds) [11:54] *** metal_cam has quit IRC (Ping timeout: 244 seconds) [12:20] *** jut has quit IRC (Read error: Connection reset by peer) [12:21] *** JSharp___ has joined #archiveteam [12:21] *** Ctrl-S___ has joined #archiveteam [12:21] *** karissa__ has joined #archiveteam [12:21] *** VonGuard has joined #archiveteam [12:32] *** VADemon has joined #archiveteam [12:36] *** mafrasi2 has quit IRC (Remote host closed the connection) [12:36] *** PotcFdk has quit IRC (Ping timeout: 506 seconds) [12:38] *** mafrasi2 has joined #archiveteam [12:42] *** PotcFdk has joined #archiveteam [12:56] *** robink has quit IRC (Ping timeout: 633 seconds) [13:11] *** atomotic has quit IRC (Quit: Textual IRC Client: www.textualapp.com) [13:16] *** scyther has joined #archiveteam [13:43] *** atomotic has joined #archiveteam [13:56] *** ete has joined #archiveteam [14:18] *** WinterFox has quit IRC (Remote host closed the connection) [14:19] *** vitzli has joined #archiveteam [15:10] *** wvdp has joined #archiveteam [15:12] *** Start has quit IRC (Quit: Disconnected.) [15:13] *** RichardG_ has joined #archiveteam [15:13] *** RichardG has quit IRC (Ping timeout: 272 seconds) [15:21] *** atomotic has quit IRC (Quit: Textual IRC Client: www.textualapp.com) [15:44] *** pfallenop has quit IRC (Ping timeout: 260 seconds) [15:45] *** Start has joined #archiveteam [15:50] *** pfallenop has joined #archiveteam [16:02] *** Froggypwn has joined #archiveteam [16:03] *** JesseW has joined #archiveteam [16:21] *** jmad980 has quit IRC (Read error: Operation timed out) [16:24] *** jmad980 has joined #archiveteam [16:29] "Is this something the Internet Archive would be interested in? From longtime wnyc dj, David Garland, .I've heard that WNYC Radio will soon remove much of the audio archive for my show Spinning On Air, which I presented there for nearly 30 years. Listen while you can! This link takes you to the archive, where you'll find many episodes, each of which has its own page. Feel free to share this link, o [16:29] r episodes that feature your favorites among my many brilliant guests." [16:30] what's the link? [16:31] http://www.wnyc.org/shows/spinning/ [16:32] sounds like something for godane :) [16:32] http://www.podtrac.com/pts/redirect.mp3/audio.wnyc.org/spinning/spinning091612.mp3 [16:32] yeah [16:36] *** JesseW has quit IRC (Quit: Leaving.) [16:36] starting to download it now [16:36] :D [16:36] godane: can you also grab everything as WARCsbesides mp3 for the audio items? [16:37] *** JesseW has joined #archiveteam [16:37] Also, we now have around 400 million URLs of fotolog saved!!! [16:37] Don't I know it. [16:37] *** JesseW has quit IRC (Client Quit) [17:05] *** Darkstar has joined #archiveteam [17:09] *** Start has quit IRC (Quit: Disconnected.) [17:09] *** atomotic has joined #archiveteam [17:12] *** roninski1 has joined #archiveteam [17:13] *** roninski has quit IRC (Ping timeout: 260 seconds) [17:20] Weirdest thing ever. I did this tool for Archive.org and now I'm experiencing like 10k visitors a day, mostly from Canada (?) [17:20] what tool? [17:20] JW_work waybackr.com [17:20] *** Guest_ is now known as eddiedean [17:20] ah, I thought that was you [17:21] Hehe [17:21] I'm trying to figure out if some botnet targeted the website or what. [17:22] It's still not very clear to me what it offers. I tried it with a minimal, one page site and it hand that back to me as a zip file — but what does it do for larger things? [17:25] Well, it is send to you via download. I mean, in your email you receive a download link. [17:26] *sent [17:28] Gotta go. I'll be later around, and we can talk :) [17:28] *** eddiedean has quit IRC (Quit: Textual IRC Client: www.textualapp.com) [17:29] *** Froggypwn has quit IRC (Ping timeout: 1208 seconds) [17:29] *** atomotic_ has joined #archiveteam [17:29] *** atomotic_ has quit IRC (Client Quit) [17:31] *** Tomcat_ has joined #archiveteam [17:31] *** atomotic has quit IRC (Ping timeout: 260 seconds) [17:46] *** Tomcat_ has quit IRC (Remote host closed the connection) [17:50] *** scyther has quit IRC (Quit: Leaving) [17:53] Guest_: are you currently doing something with flickr and the wayback machine api> [17:53] ? [17:53] eddiedean ^ [17:54] *** RedType has joined #archiveteam [17:54] oh, eddiedean is gone [17:55] *** jut has joined #archiveteam [17:56] *** Froggypwn has joined #archiveteam [17:57] *** Ymgve has quit IRC () [17:58] *** vitzli has quit IRC (Leaving) [18:05] *** philpem has joined #archiveteam [18:23] *** SN4T14 has quit IRC (Remote host closed the connection) [18:26] *** yipdw_ is now known as yipdw [18:28] *** SN4T14 has joined #archiveteam [18:34] *** JW_work has quit IRC (Read error: Operation timed out) [18:35] *** vOYtEC has joined #archiveteam [18:35] *** schbirid2 has quit IRC (Remote host closed the connection) [18:40] *** metalcamp has joined #archiveteam [18:40] So, I am actively and aggressively downloading the Spinning on Air archives. [18:40] Only downside, if you want to call it that, is that it is REALLY a slow download. 100k/s [18:40] So, 8 minutes a podcast. 50mb apiece. [18:41] *** JW_work1 has joined #archiveteam [18:42] *** JW_work has joined #archiveteam [18:45] *** JW_work2 has joined #archiveteam [18:46] would more download streams help, or are they just straining their upstream? [18:46] SketchCow: i'm also downloading them [18:46] *** JW_work has quit IRC (Read error: Operation timed out) [18:47] *** JW_work1 has quit IRC (Ping timeout: 370 seconds) [18:48] *** JW_work has joined #archiveteam [18:48] *** JW_work2 has quit IRC (Read error: Operation timed out) [18:50] *** Tomcat_ has joined #archiveteam [18:53] *** ohhdemgir has joined #archiveteam [18:59] uploaded: https://archive.org/details/wnyc-spinning-on-air-08-14-2005 [19:00] *** ete has quit IRC (Read error: Operation timed out) [19:02] *** LastNinja has joined #archiveteam [19:02] I'm back - been away for a few days and connection dropped out [19:06] welcome back [19:07] *** schbirid has joined #archiveteam [19:21] *** Start has joined #archiveteam [19:24] WB [19:26] SketchCow I'm getting 2.3MB/sec from that site [19:30] *** Ymgve has joined #archiveteam [19:37] *** roninski1 has quit IRC (Ping timeout: 260 seconds) [19:39] *** roninski has joined #archiveteam [19:44] Woo hoo [19:45] godane: It's good you're grabbing them, but can you ensure all the metadata gets in there? [19:49] *** bwn has quit IRC (Ping timeout: 246 seconds) [19:51] yeah I don't know how to do proper manual grabs of things :( I just make the scripts go, lol [19:53] godane: and please also do them as WARCs!!! [19:59] SketchCow: are you able to increase rsync connections to FOS? [20:00] Fotolog is really slowing down due to the 100 connections limit [20:00] And they should have shutdown quite some time ago, so I'm not sure how long they'll be up [20:00] Or can FOS not handle that at the moment? [20:04] *** Tomcat_ has quit IRC (Remote host closed the connection) [20:05] arkiver we've been talking about FOS being slow for days :( [20:05] I stopped my gametracker & wiki grabs, because it's just beating on FOS more [20:05] I stopped them 4 days ago, and they are still trying to push out the pidly 20G that was stacked up on them [20:08] You can cancel all that then [20:09] I'll just requeue your items [20:09] fotolog really is a lot more important [20:09] no I'm letting them flush, there's only 1 left [20:09] it's just the last 1 of them that's finishing [20:09] arkiver: are we using FOS as the only rsync target for fotolog? [20:09] currently yes [20:09] but from what I've seen and other people are saying, nobody can push more than 100-150KBytes/sec to FOS [20:10] arkiver: If we had another rsync box, that could possibly alleviate some of the headache, right? [20:11] definitely [20:11] Atluxity: Still around? [20:11] who owns fboyd? [20:11] Fletcher [20:11] ok [20:11] Fletcher: I still have your target in the fotolog grab (deactivated) [20:12] Can we activate that for the WARC parts of the fotolog grab? [20:12] *** kris33 has joined #archiveteam [20:13] *** Microguru has joined #archiveteam [20:18] I'd spin up fotolog right now on 2 systems, but without an RSync target besides FOS, I'm not really doing anything usefull, and just slowing everyone else down [20:19] phuzion: hanging in here... [20:19] if everyone that's here, could stop everything but fotolog that would help [20:19] ErkDog, are they long term instances? [20:19] beacuse then only fotolog would be going to FOS [20:19] yes I can run them indefinitely [20:19] *** kris33 has quit IRC (Textual IRC Client: www.textualapp.com) [20:19] *** bwn has joined #archiveteam [20:19] hop in #newsgrabber [20:19] we can put them to use [20:19] Atluxity: You hosted an rsync target at one point, didn't you? [20:19] http://tracker.archiveteam.org/ftp/ [20:19] I can push and pull some damn packets [20:19] just have to have somewhere to push them [20:20] I've done 2.1TB of FTP in less than a week bros [20:20] phuzion: no, I got a host ready to do so, but never got around to set up the rsync part [20:20] ErkDog, newsgrabber doesnt use FOS [20:20] lol you are trying to solicit me for newsgrabber ;) [20:20] yes [20:20] phuzion: also, I am unsure about the speed available, that can be... interesting [20:20] is there an e-Penis I mean tracker for it yet? [20:20] not yet, its done differently [20:20] awwwwwwww [20:20] Atluxity: How much disk do you have available? [20:21] arkiver, we need an "e-Penis tracker" [20:21] phuzion: 10TB [20:21] HCross: Leaderboards need to be represented by ascii dongs. 8=======> longer the dong, more data you've transferred. [20:21] SWEEEEEEEEEEEEEEEEEEEEET [20:21] * ErkDog agrees [20:23] Atluxity: Are you willing to use that 10TB as an rsync target? I could possibly help you out a little bit with the setup [20:24] phuzion: I would be glad to give you access to it to help me out [20:25] I am not quite sure how well monitored the raid is, now that I come to think of why I did not go ahead last time I had time [20:25] :\ [20:25] Awesome. I don't know what needs done for the megawarcing and whatnot, that's something that we'd need someone else's input on [20:25] Oh [20:30] phuzion: I got word the raid is monitored [20:31] ok [20:31] add him as target for wiki & gametracker & fotolog [20:31] so FOS can breath, lol [20:31] eeeeek [20:31] It might be better to dedicate Atluxity's server to fotolog since that's most at risk right now [20:32] That's what I was thinking. [20:32] (MrRadar's idea) [20:35] so use the new untested thingy to handle the most endagered stuff? [20:35] *** RedType has quit IRC (Read error: Operation timed out) [20:35] how bad can it be? [20:36] whats the worst that could happen... [20:36] *** chfoo has quit IRC (Read error: Operation timed out) [20:36] *** ersi has quit IRC (Read error: Operation timed out) [20:36] *** Smiley has joined #archiveteam [20:36] phuzion: ssh pub key to me privmsg, and I can give you access [20:36] *** Jonimus has quit IRC (Read error: Operation timed out) [20:36] *** SmileyG has quit IRC (Read error: Operation timed out) [20:36] I need to go to bed soon [20:36] *** aMunster has quit IRC (Read error: Operation timed out) [20:36] *** rossdylan has quit IRC (Write error: Broken pipe) [20:36] *** beardicus has quit IRC (Read error: Operation timed out) [20:36] *** nwf has quit IRC (Read error: Operation timed out) [20:36] *** sivoais has quit IRC (Read error: Operation timed out) [20:37] *** chfoo has joined #archiveteam [20:37] *** sivoais has joined #archiveteam [20:37] *** MMovie has quit IRC (Read error: Operation timed out) [20:37] *** lbft_ has quit IRC (Read error: Operation timed out) [20:37] *** Mayeau has quit IRC (Read error: Operation timed out) [20:38] *** vegbrasil has quit IRC (Read error: Operation timed out) [20:38] *** oli has quit IRC (Read error: Operation timed out) [20:39] *** lbft has joined #archiveteam [20:39] *** oli has joined #archiveteam [20:41] LOL Atluxity :-D [20:42] *** closure has quit IRC (Read error: Operation timed out) [20:42] *** ersi has joined #archiveteam [20:43] *** Asparagir has joined #archiveteam [20:44] *** Start has quit IRC (Quit: Disconnected.) [20:46] ErkDog :) [20:46] I guess having it and maybe something going wrong is better than not having it at all because FOS is overloaded ;( [20:46] *** VADemon has quit IRC (Quit: left4dead) [20:47] ofc [20:47] *** mhazinsk has quit IRC (Ping timeout: 633 seconds) [20:47] but this can look a bit like a ddos [20:47] and if the speed is any throttled, which is untested, then it will have the same effect [20:48] *** RedType has joined #archiveteam [20:50] hmm... someone sais the uplink is 100mbit :\ [20:50] thats residential speeds [20:51] no [20:52] well -decent- firewalls would see that it's legitimate traffic, I wouldn't say transfers from 20-30 locations would be a "ddos" per se [20:56] 20-30 locations? I run 50 myself [20:57] well that's probably why FOS is shitting all over itself [20:57] ErkDog is what happens when you anthropomorphize pv(1) [20:57] *** ete has joined #archiveteam [20:57] if ONE person is pushing 50 rsynch's to it [20:58] like do you know how harddrives work? and storage systems? lol [20:58] yipdw: ha [20:58] Atluxity: is it you who's aiming the proton cannon at fos? [20:58] ErkDog: if we want less pipelines, we can throttle the tracker? [20:59] well true, but the problem is, I don't think anybody has any real way of seeing how many incoming rsynch's there are [20:59] xmc: I think I might be 50% of the reason... [20:59] and whether it 'needed' to be throttled [20:59] and since there are half a dozen different trackers sending jobs in [20:59] I know how many there are, it's not really relevant though if fos needs work [21:00] I trust ppl to let me know when to hold back [21:00] well the fact remains, that any given system can only take X number of RSynch's before you end up with 1/Y performance per Rsync, where Y is a number less than 1 [21:00] well Atluxity we've been saying for days how FOS is getting pounded in the face [21:01] I wish I could increase performance with more rsync processes [21:01] that'd be fucking awesome [21:01] I mean greater than one [21:01] Atluxity assuming no bottlenecks on bandwidth, so say, a 'lan' connection, 1 Rsynch will finish 50 units of work, faster than 50 rsync's will finish 1 unit each [21:02] huh [21:02] anyway, fotolog is already at 60 requests/minute, I can push it down a little more and we'll see if it makes any difference [21:02] Hmm [21:02] It's not big items [21:02] I suspect the answer will remain "fos needs work" and we will know when it's fixed because SketchCow will tell us it's fixed [21:02] FTP is not going to FOS [21:02] I know FTP isn't [21:02] GameTrailers can be paused. [21:02] we are talking about fotolog, wiki & gametrailers [21:03] But fotolog is important, do not slow it down, slow other things doen [21:03] down* [21:03] specifically targets to Fos [21:03] GameFront is paused [21:03] there isn't much else to slow down, really [21:03] Atluxity if you have 50 workers going, then you are sending the haddrive to 50 different places all at the same time, whereas if you had 1 worker going it would send one stream of data [21:03] rsync traffic to fos is fotolog and gamefront [21:03] and Wiki [21:03] ErkDog: calm down [21:03] wiki goes to FOS [21:04] And wiki is just really small [21:04] ErkDog: I got 50 hosts, running with a concurrency of 10 each... [21:04] root@teamarchive0:~# lsof 2>&1 | grep rsync | grep warrior | grep wiki [21:04] rsync 5063 1001 cwd DIR 9,1 12288 77596257 /1/CHFOO/warrior/wikis/Floxit [21:04] rsync 5077 1001 cwd DIR 9,1 12288 77596257 /1/CHFOO/warrior/wikis/Floxit [21:04] rsync 5077 1001 5u REG 9,1 543424512 77594660 /1/CHFOO/warrior/wikis/Floxit/.wikis-mediawikieu_hu.assassinscreed.wikia.com_api.php_hu.assassinscreed.wikia.com_wiki_-20160310-163250.warc.gz.rsZfI1 [21:04] Like really, not a lot is going to FOS [21:04] well Atluxity said he didn't really understand what I meant [21:04] seriously stop telling me what I can find [21:04] it's not warrior projects I think [21:04] kthxbai [21:04] it's fotolog and if we can't slow it down then we'll just wait for fos to spring back [21:04] this isn't fatal [21:05] Then we'll wait for FOS to spring back [21:05] we might also be able to use Kenshin's stuff but I think he is still rebuilding the disk array on that one [21:05] all I was trying to explain to Atluxity since he's running so many hosts, that if he's downloading faster than he can upload to FOS then having more hosts results in diminishing returns, as rsync to an overloaded target becomes the bottleneck [21:05] YES WE KNOW [21:06] jfc [21:06] well he didn't seem to know, because when I explained it he said "huh" [21:06] ErkDog: I hate to be a debbie-downer here, but you are not telling us anything we don't know. [21:06] and "I trust people to ask me to slow down" [21:06] we have literally known for __MONTHS__ that FOS is underpowered. [21:06] *** ErkDog was kicked by xmc (learn to keep your mouth shut a bit more) [21:06] *** vegbrasil has joined #archiveteam [21:06] *** ErkDog has joined #archiveteam [21:06] I think that escalated too quickly [21:06] *** Asparag-1 has joined #archiveteam [21:06] zino__: I'm afraid we need to use your target again [21:07] Would you have some rsync space available for fotolog? [21:07] zino__ has an rsync target? [21:07] Yeah [21:07] oh cool [21:07] arkiver: Okidokey then. :) [21:07] *** Asparag-1 has left [21:07] He still holds 12 TB of gametrailers [21:07] So we need to move that off still [21:07] yipdw: would you be able to move that 12 TB to IA? [21:07] well every time I've mentioned FOS someone tells me it's my problem and to provide traceroutes [21:07] I could, I don't have access to itthough [21:08] *** xmc sets mode: +o yipdw [21:08] *** xmc sets mode: +oo chfoo- chfoo [21:08] *** xmc sets mode: +o swebb [21:08] *** swebb sets mode: +o edsu [21:08] *** swebb sets mode: +o ersi [21:08] and I go 'but everyone can only push 100KBps' and then nobody says anything else about it, lol [21:08] ErkDog: we've heard you loud and clear [21:08] if zino__ has space then it'll help a lot of things [21:09] fair enough, also, to be fair, I still don't know many of you or what your roles in this team are.... the only thing I've been able to skry so far is that SketchCow works for archive.org and is in charge of FOS [21:09] loudest voice wins [21:09] LOL [21:09] Status page says I have 14T left, but still expecting .8T from the ftpgrab. See http://eldrimner.lysator.liu.se:8080/archiveteam.txt [21:09] loudest voice wins, not everyone gets +o [21:09] ohhhh zino__ is eldrimner you? [21:10] Yep [21:10] I've been sending arkiver all kinds of queue's to add :-D [21:10] Does anyone know of a good way to make sure a CD I'm archiving can be faithfully copied with dd as an ISO or whether I need to copy it as a BIN+CUE? [21:11] johtso: is it multisession or does it have more than one track => bin/cue [21:11] zino__: let's get the gametrailers stuff off soon. I hope yipdw is able to do that [21:11] I see betadump and friendsreunited too [21:11] arkiver: if you have shell access, you can run the megawarc factory also [21:11] xmc: is there a simple way to see what tracks a cd-rom has? [21:11] or zino__ can [21:11] yipdw: ok, will do [21:12] If someone points fotolog @ eldrimner let me know, I'll spin up instances [21:12] I never used it though, so will need to figure some things out first [21:12] just make sure your S3 keys are granted upload access to that collection [21:12] ok [21:12] johtso: i don't have a good way, sorry [21:13] arkiver: betadump (one of the empty ones) was created in case SketchCow needed it. Just in case. [21:13] I see [21:13] so fotolog is ready? [21:14] (I see the dir there) [21:14] Fotolog never got directed towards me, so also empty. :) [21:14] The ones that show 4.0K have never been used. [21:14] So, who here knows exactly what needs done to get an rsync target running? [21:15] Megawarcing and whatnot. [21:15] * yipdw does [21:15] *** beardicus has joined #archiveteam [21:15] the README in archiveteam-megawarc-factory is pretty good [21:16] alard as usual was very meticulous [21:16] yep [21:17] arkiver: Misread you there I think, yes, fotolog is ready to receive if you need it. rsync module "fotolog". [21:17] I set it up last time there was some throuble and never removed it. [21:17] ok [21:17] I also PMed you about it [21:17] but will activate it now [21:18] *** MMovie has joined #archiveteam [21:18] Yea, but client isn't really good at showing me when I get PMs... [21:19] right, I'll let you know in this chat next time [21:19] FOS is deactivated for fotolog! [21:21] *** schbirid has quit IRC (Quit: Leaving) [21:23] ErkDog: Pointing done. Do your thing. :) [21:25] thanks :-D [21:30] It looks like the website is a lot faster then it was a week ago [21:42] *** jut has quit IRC (Quit: jut) [21:44] bunch of stuff probably pushed up into cloudflare now, lol [21:44] zino__ what's your HDD setup like for that 28T array? [21:45] *** Ymgve has quit IRC () [21:47] ErkDog: 12xSpinning rust + LSI Megaraid. Probably close to 5 years old by now. [21:47] *** Ymgve has joined #archiveteam [21:48] chfoo: can you send me the logs for fotolog? Especially the log from today [21:48] Turns out fotolog gives an 'over capacity' message as 200 [21:49] ErkDog: Dell R515. [21:49] arkiver: Well, that's not nice. :-/ [21:49] yeah, fixing that in the script now [21:50] spinning rust, lol [21:51] Raid 5 or 1+0? [21:51] looks like wnyc.org is down or not displaying pages [21:51] *** closure has joined #archiveteam [21:52] http://puu.sh/nBZNs/31c9709137.png [21:52] loads for me [21:53] ErkDog: Raid 5+0 if I remember correctly. [21:53] 5+0????' [21:53] ohhhhhhhhhhh [21:54] that's an interesting model but also explains the performance [21:54] You mean the slowness? :) [21:54] well no because the 0 part gives you more speed across the individual blocks [21:55] woop woop woop off-topic siren [21:55] take it to #archiveteam-bs [21:55] this is supposed to be a low traffic channel [22:04] *** MMovie has quit IRC (Read error: Operation timed out) [22:04] *** sigkell has quit IRC (Ping timeout: 260 seconds) [22:05] *** aMunster has joined #archiveteam [22:06] *** MMovie has joined #archiveteam [22:07] *** Jonimus has joined #archiveteam [22:07] *** swebb sets mode: +o Jonimus [22:09] as well as a low-derp channel [22:14] *** sigkell has joined #archiveteam [22:14] *** JW_work has quit IRC (Quit: Leaving.) [22:16] *** WinterFox has joined #archiveteam [22:27] *** ndiddy has joined #archiveteam [22:27] Best way of getting a copy of a phpBB forum into the archive? [22:27] http://www.astra2forum.com is going away on the 14th [22:28] archivebot with appropriate ignores [22:28] or, wget-warc-lua with the phpbb script [22:28] though i think that's kind of fiddly [22:28] *** nwf has joined #archiveteam [22:29] 4 days is a really stupid notice [22:29] *** Mayeau has joined #archiveteam [22:30] *** Sk1d has quit IRC (Ping timeout: 250 seconds) [22:31] *** mhazinsk has joined #archiveteam [22:31] it's a pretty small site [22:37] it has a link to Lycos on it, thats how old it is [22:47] *** atomotic has joined #archiveteam [22:52] *** atomotic has quit IRC (Client Quit) [23:11] *** plog99 has joined #archiveteam [23:13] *** fpoee has quit IRC (Ping timeout: 250 seconds) [23:31] *** wvdp has quit IRC (http://www.kiwiirc.com/ - A hand crafted IRC client) [23:36] *** Stilett0 has joined #archiveteam [23:37] *** JSharp___ has quit IRC (Ping timeout: 260 seconds) [23:37] *** JSharp___ has joined #archiveteam [23:38] *** Stiletto has quit IRC (Ping timeout: 260 seconds) [23:51] *** Stilett0 has quit IRC (Read error: Operation timed out)