[00:09] so where can I put another shareware CD since batcave is being retired/retired? [00:22] https://drive.google.com/ Hmmm! [00:22] nitro2k01: what could possibly go wrong [00:25] That about sums up my thoughts [00:32] Just watched this [00:32] http://www.amazon.com/Comic-Con-Episode-IV-Fans-Hope/dp/B007RV6NJW/ref=pd_sim_b_1 [00:32] SO GOOD [01:13] dashcloud: either upload straight to archive.org or I can give you an ftp location [01:15] ftp location would be good- thanks! [04:04] dashcloud: Just upload it to archive.org. [04:04] archive.org/upload [04:04] Awesome, he's gone [04:04] We should just be shoving CDs into archive.org and we add them to the collection. [04:07] I got it from him, it's up [04:08] http://archive.org/details/cdrom-best-of-kid-games [04:09] SketchCow: fyi, progress :) [04:15] hmm did archive.org just shit the bed [04:24] i'm still uploading to it [05:06] Always good to know. [05:37] some guy tells me http://www.bladeandepsilon.com/ might be going down [05:44] what a super site [07:19] The batcave shove-off continues. [07:24] only 20 MB/s :p [07:41] We are absolutely going to fill FOS tonight. [07:41] The machine is just thrashing like hell. [07:41] Too much Mobileme. [07:41] It's technically infeasable for it to even just .tar and upload these files fast enough. [07:43] why is SketchCow's name highlighted in http://badcheese.com/~steve/atlogs/ ? [07:44] Yeah, just checked. [07:44] We're adding 50gb to that machine every 40 minutes. [07:45] So basically, let's go with "about a gig a minute" [07:45] It takes longer than that to .tar it up [07:45] And takes a little less to upload it. [07:45] Disk is at 96 percent. [07:46] I think I need alard to stop the uploading via FOS for a day. [07:47] slow for a bit to let it get a head start? [07:47] Is there any way we could offload some of the menial processing to the clients in the future? [07:47] There's not really any snow. [07:47] slow [07:47] Well, the right thing to do is upload directly into archive.org, but it can't easily be done with smaller clients. [07:47] No, the problem is that MobileMe is a massive massive set of data. [07:47] I bet THE MAJORITY OF COMPANIES ON EARTH don't transfer this sort of data. [07:48] Can you physically ratelimit your uploading to allow the machine to be slightly ahead? [07:48] Your solutions are full of assumptions [07:49] To be honest, I don't know why the archive.org admins haven't flipped out over what's going on here. [07:49] We must be demolishing some major internal structures. [07:51] Well, I have to go to bed soon. [07:51] We've filled FOS. [07:51] It's going to error out tonight. [07:51] Probably in, oh, 2-3 hours. [07:51] No way to stop it other than to kill rsync. [07:51] I hope the uploader can handle that. [07:52] I see some of these 50gb hero sets are taking 2-3 hours to turn into .tar files. [07:52] That's precious. [07:54] I really fucking hate mobileme. [07:54] So huge. [07:54] So much shit. [07:54] I mean, after this, we'll be able to do ANYTHING. [07:54] So I guess that's good. [07:55] OK, that's it. Killing it now. [08:01] All gone, going to let this system catch up for a day [08:44] * SmileyG ponders [08:44] SketchCow: random question, but should "I" as random uploader, stop? [08:45] or was this the "alturnative" route for privildged users which just stopped? [09:01] SmileyG: You've been doing tiny amounts of traffic compared to the people in this list: http://memac-tamer.heroku.com/ [09:02] Hey, is anyone else having problems rsyncing to fos.textfiles.com ? [09:03] pronoiac: Jason killed rsyncd because we filled it up. [09:03] It needs some time to digest what we've got. [09:04] Cool. [09:04] Is there a way to report problematic items? [09:05] Or are they automatically handled, with the stragglers getting retried after the first pass? [09:06] Also, holy crap, some people are running dozens of crawlers. Wow. [09:08] Yeah, I'm not sure how no one has noticed oli's 17TB of traffic on like...three IPs over the last month. [09:08] And kenneth is...I just don't even know. [09:09] Those are using the S3 interface? So, each process has around 1.5 megabytes/sec of bandwidth? [09:12] It's - I don't think I have that much bandwidth to my router. In the next room. [09:15] kenneth is using heroku resources and the s3 client [09:15] i think oli is also using the s3 interface [09:15] those with high throughput are supposedly using the s3 interface directly [09:16] 2-3 hours to just .tar or tar.gz? [09:22] Oh, I meant his collective bandwidth >> my measly non-gigabit ethernet. [09:22] i think oli's using encyclopedia dramatica's servers [09:23] and heroku is hosted within amazon's ec2 [09:23] Hi, I'm new here. What's going on tonight? [09:24] keeping a watch out for trogdor [09:25] What is the current project? [09:27] Also, am I correct to guess that this community first arose around the Reocities project? [09:28] archiveteam was one of the groups that went after geocities [09:28] I can't speak for archiveteam - I'm just doing some mirroring of MobileMe. [09:29] ahh, okay [09:29] the current big project is mobileme, but the upload staging host is full at the moment and needs some time to get cleared out [09:30] Burninate: Reocity is someone else [09:30] (the clearing is underway, but takes some time, even with LAN network speeds) [09:30] Burninate: AFAIK [09:31] Has the legality of all this ever been challenged, either by copyright or privacy concerns? [09:32] lol, legality [09:32] the house is burning. who has time to argue about legality? [09:32] Assholes, that's who [09:32] I'm curious if you've ran into any [09:32] nopes [09:32] besides, who should they bicker at? We're a bunch of individuals [09:33] some of the stage6 content might get challenged and go dark [09:33] shit, you go that far back? [09:33] I liked stage6 [09:34] but it will still exist for historians to examine [09:34] no. that was a personal project of mine, prior to archiveteam [09:35] sadly, i only got metadata for about 4.24% and video files for about 1.17% of what was there [09:35] ... which still amounts to 304GB [09:35] that's a lot more than 0.0% [09:36] it's also a little skewed toward what i was personally interested in :-\ [09:38] but it's going up at http://archive.org/details/stage6/ [09:39] Dumb question: Do you know where people are doing the mirroring? Home, VPS, dedicated servers, ? [09:39] all of the above [09:39] whichever they have available [09:40] answer is no [09:41] I'd be curious to see something like TOR's "Good & Bad ISPs" page, for this. [09:42] Rather, I'd like that. [10:03] Wyatt: yeah I realise i'm quite insiginificant ;) - I meant is it worth me downloading if its just gonna die when it tries to rsync it back up? [10:04] seesaw's rsync should keep retrying until it succeeds [10:04] ^ [10:05] ah ok [10:08] Ugh, I just found another loop of some sort. [10:08] It's retrying resources that returned 404s. [10:09] 195 times and counting. [10:09] the same resource? [10:10] I grepped for 1 specific entry. [10:10] There are many files with 404s in the wget log. [10:10] if different pages refer to it, wget will try on each new page that refers to it [10:11] dead links are unfortunately the ephemeral nature of the world wide web [10:13] This might be the cause of a memory problem - it's using 1.7 gig of memory right now. [10:15] Since I started on the MobileMe mirror, it's the second problem I've seen that, with the structure we have, can only be fixed in wget. [10:16] yeah, with a large recursive get, wget will take a lot of memory [10:16] * Coderjoe slinks into bed [10:17] Er, yeah, it's late. [10:18] Oh, alternately: when it gets stuck in a loop, get a list of the resources it's fetched, then do another non-recursive crawl using just the files on that list. [10:22] Coderjoe: smart. [12:17] Greetings all! [12:27] hello [12:28] hi dnova [14:57] erm [14:57] bug: If you've done touch ./STOP, it doesn't retry the rsync (I dunno if it does anyway)... [15:00] hense the asking this morning :( [15:00] ./STOP != ./stop [15:01] oh, never mind, caps in original [15:01] yah [15:01] it STOPPED, it didn't continue to retry and then stop. [15:01] * SmileyG notes he has 10gb of "stuff" in ./data/ [15:01] not sure how much of it actually uploaded tho :S [15:03] Sweet. I got a bunch of free aws credit. I think I'll do that project gutenberg >> github project I've been wanting to do [15:04] Morning. [15:05] morning jason [15:05] Hey SketchCow [15:05] mornin' [15:19] morning, SketchCow [15:19] you saw about the progress, right? :) [15:19] Mornin. [15:20] I presume theres some way to just "upload the finished stuff" ? [15:21] balrog_: Are you discussing #discferret in #archivteam again? [15:21] maybe, I'm in too many channels for my own good. [15:22] * SmileyG realises hes turning into that loud annoying little kid :/ [15:22] Keep that right. [15:22] SmileyG: Someone has to be, underscor is now 18 [15:22] hmmm I'm 26? :) [15:22] Mental age of..... about 12 I think. [15:22] Shhh, don't tell your classmates that [15:23] hmmmm 28 [15:23] ¬_¬ [15:23] I stopped counting my age too, and now I constantly forget how old I am :D [15:26] Yeah, I've got us down to 90% already (from 96%) and all the uploads are going much faster. I think we defeinitely need to call today No MobileMe day [15:26] But it returns one day later. [15:26] http://business.fortunecity.com/analysis/885/ d'awww [15:26] Is this "offically" part of your job SketchCow btw? [15:26] * SmileyG is still confused by the whole situation. [15:27] talking of which i have like 4000 screenshots to crop (some random shit on the bottom and edge of each screenshot) [15:27] winr4r: imagemagik ? [15:27] SmileyG: yes [15:27] Once we finish MobilMe, we can all sit back and enjoy a nice dinner of Meat Goats [15:27] :) [15:27] Hey, happy birthday underscor! [15:28] SmileyG: problem: most of them are on Wyatt's VPS and i'm wondering if his host will get sad if i totally rape the CPU for like 20 minutes straight [15:28] winr4r: ah. [15:28] Wyatt: talking of which, prod me when you're around [15:28] winr4r: hmmm I might know who hosts that VPS [15:28] Let me go see. [15:29] about to do a test run on 500 of them here [15:30] any good VPS 1. wont notice, 2. won't care, 3. Will have limited anything worth limiting anyway. [15:32] I wonder if these 28K ./data/s/s_/s_i/s_igusa/web.me.com/files/web.me.com/s_igusa/能勢川バイブルキャンプ [15:33] I did actually figure out what happened. [15:34] After the FOS reboot, I didn't have the daemon running that makes the new 50-gb chunks. So we ended up with, like, 5tb of data that had to then be dealt with. [15:34] Ouch. [15:34] And then we started to hit lag on them, because now the machine was trying to catch up AND take new data. [15:34] * SmileyG powers up the mega-super-extra-zero-II-splittermon [15:37] oh ffs, bad news is that the screenshots have about 15 pixels randomly missing from the right hand side after random-shit cropping [15:37] oh well, i have 4 days! [15:37] * winr4r headdesks [15:37] :< [15:40] http://i.imgur.com/Jktyl.png [15:40] which might seem minor but it's not faithful at all to the original [15:43] the banner at the top :« [15:44] * SmileyG passes some spare ] to winr4r [15:48] :/ [16:05] aggro: How's Parodius going? [16:07] winr4r: For what it's worth, I find graphicsmagick can sometimes be a bit more reliable than imagemagick. (Assuming the cropping wasn't a capture-end thing.) [16:08] mistym: i've fixed the bot so that it doesn't have that random junk on the sides [16:08] running it again, seems to work better [16:08] :D [16:09] (if i don't get all 4000-ish by closing time i'll be okay since i have the others, even if they are sub-optimum) [16:09] (but i see no reason that will happen) [16:20] I wish we had Google Video records [16:20] I'm wondering if mobileme is bigger than that now [16:20] I mean, what we got [16:20] I think Google Video is easily above 200 TB in its entirety [16:20] (maybe) [16:21] oh, wait [16:21] duh [16:21] http://archiveteam.org/index.php?title=Google_Video [16:21] yup [16:21] 18 TB, shit [16:21] there's two people on the mobileme grab who have done more than that [16:21] yeah [16:21] I suspect oli will be exceeding that today too [16:24] heh [16:24] 2012-04-26 11:24:16 URL:http://gallery.me.com/emewsiq/100238/DSC_0628/large.jpg [651059/651059] -> "/dev/null" [1] [16:24] that reminded me that Nikon is now making cameras with 36 MP sensors [16:24] one raw capture off one of those is around 48 MB [16:24] have fun backing that shit up [16:25] i'd like to see a lens resolve that well [16:25] I don't know of any [16:27] yipdw: i make it 63 mb at 14-bit NEF [16:27] for* [16:27] even better :P [16:27] though i might have miscalculated because that's pre-bayer [16:27] * winr4r shrugs [16:28] NEFs (and CR2s etc) are also losslessly compressed in some fashion [16:29] yipdw: now i think about it, your original figure seems right, so meh [16:29] yipdw: Depends on which Nikon raw format. Some older ones actually had lossy compression. [16:29] but, yes, the best resolution I've seen out of a camera system that I own is an EF 100mm f/2.8 macro on a 5D Mark II [16:29] I'm not sure if the 100mm is outresolving the sensor [16:29] frankly I don't give a shit because the results are insane [16:29] mistym: they had the balls to call that raw? [16:30] ^ [16:30] yipdw: Yeah. Lemme grab a cite. [16:30] I seem to recall they cut that out later, but at least at one point in time they were definitely dropping some data. [16:30] mistym, yipdw, as i recall that was always optional [16:31] http://www.completedigitalphotography.com/?p=489 [16:32] mistym: oh, that's not as bad as I thought [16:34] Still seems pretty cheeky to call it "raw" when it's had a lossy curve applied. [16:34] maybe [16:34] it sounds like they called it "compressed raw" [16:35] which isn't really a misnomer, especially if Nikon isn't calling it lossless; but, yeah, I can see how that can be misleading [16:35] That's true. [16:35] i like how everyone actually cares about that [16:35] I have a vague memory of some camera either defaulting to it or not offering the option, but clearly hasn't been the case in awhile. [16:35] I know cRAW seems to be a trend [16:36] I'm really not sure why [16:36] People want to be cool and use raw, but don't actually want to commit. [16:36] it will give you higher shooting rates, but [16:36] well [16:36] I can actually think of one use case, but I'm not sure it makes any sense [16:37] If you want high shooting rates, not sure cRAW is acceptable even then. Probably still too pokey. [16:37] say that you want to maximize shot rates but want to preserve the ability to manipulate white balance in post [16:37] I dunno [16:37] it's a weird combo [16:37] might as well get the WB right off the bat [16:38] and if you want high shot rates for stills, then why are you not using something like a 1D Mark IV :P [16:40] eh [16:40] 8 frames per second in raw fills up memory cards really, really fast [16:41] as in minutes [16:42] I guess if you're doing that, you might as well have a file transmitter [16:43] yipdw: as in carry a laptop around to make yourself even more of a theft target? [16:43] winr4r: wireless file transmitter [16:44] I mean, the laptop doesn't have to be around with you [16:44] just have a bandolier of memory cards [16:45] good luck with that at 48mbit/second *on a 4.1 megapixel camera* [16:45] http://www.usa.canon.com/cusa/consumer/products/cameras/consumer_cameras_wft/wireless_file_transmitter_wft_e7a pulls it off [16:45] sorry, 48 mega*bytes* per second [16:45] I mean, no, it will not handle a continuous stream of raws [16:46] but then again neither will the camera's internal buffers [16:46] so it's not really worth considering that case [16:54] in any case, over 500 screenshots, about 3000 to go :) [17:23] shaqfu: working on a slow crawl of the hosts atm with the exception of nesdev, which Tepples has said he is archiving himself (http://archiveteam.org/index.php?title=Parodius_Networking). Demiforce and donut both require authentication, so no archive there. Currently on spliceworld. The hosts aren't taking long at all. [17:23] (Even with the delay) [17:24] Looks like transcorp has a small forum. That might take a while with the delay, but otherwise going fine right now. [17:25] Also, I'm considering updating the wiki article. Judging by the he-said she-said on their forum I think we've miscommunicated with each other. [17:29] aggro: What list of URLs are you using? [17:32] www, ffxiquest, gaijin, haven, hd, jdc, ps2pc, pt, sl1me, spliceworld, transcorp, zatos [17:32] example: haven.parodius.com [17:33] That's probably most of it. Do you want the list I compiled? I think there are a few more [17:34] Sure. I haven't examined the other sites they host (like rocknes) [17:35] Give me a few minutes - starting that box up [17:38] aggro: donut, chojin, sdlmame, mv, blargg.parodius.com/gme_java/, soulstream, dowd, vice, and maybe all the stuff under koitsu.org [17:39] Oh, right, strikeout donut [17:39] donut asks me for auth --right. thanks. [17:40] There might be a few more I missed, but I was at the point where I was finding a lot of placeholder sites [17:41] cool. I'll add them. [17:44] Thanks again [17:50] 1 [17:51] hm [17:51] I wonder if it's bad form to use internet memes in commit messages [17:52] I just threw # encoding: utf-8 on a shitload of source files and the urge to write ENCODE ALL THE FILES as the commit message is hard to resist [17:52] I think that's a perfect commit message. It's probably shorter than what you were going to write too :P [17:55] actually, maybe I can base64 http://www.quickmeme.com/meme/3ozhsp/ into it [18:23] fos.textfiles.com having issues? [18:24] it's full, rsync halted [18:27] Slightly off-topic, does anyone know the URL of that list of hard drive prices? [18:27] http://edwardbetts.com/hdd_price_per_tb/ that one? [18:27] http://edwardbetts.com/price_per_tb/internal_hdd/ [18:27] That's the one; thanks [18:27] oh bummer, beat to it [18:28] winr4r: your link was better :) [18:28] ersi: maybe not, because external ones are still cheaper per terabyte, oddly [18:28] actually, this one is better, ie cheaper :) http://edwardbetts.com/price_per_tb/external_hdd/ [18:29] yeah [18:29] tasty tasty terabytes, growing some shells for protection [18:29] huh, $170 for 3TB [18:30] I should pick up five of those and add those to my home disk array [18:30] that'd like double the size of the fucking thing [18:30] indeed you should [18:30] did the IA buy those? Seagate FreeAgent GoFlex Desk 3TB USB 3.0 Desktop External Hard Drive [18:31] pberry, you're not on the charts so I can't know: how fast are you downloading mobileme? [18:32] it's usually suggested to use seesaw-s3 only if you have at least 1.5 MB/s (up and down), but maybe if fos is down this can be lowered a bit [18:34] (so that you upload directly to archive.org) [18:34] I'd just wait it out [18:34] seesaw-s3 isn't a drop-in [18:37] Nemo_bis: just having trouble uploading [18:37] rsync is very grumpy [18:37] ah [18:37] just read back up... [18:37] thanks [21:07] underscor, ping [22:36] seesaw-s3? [22:47] Has anyone downloaded the Dell documentation library? [22:52] Seems straightforward enough, but if someone already has it, I'd rather not duplicate effort