[01:01] yipdw: wow. that article shows a gross misunderstanding as to how that bill was created [01:08] oh [01:08] i just noticed who the author was [03:38] i found out that there was a stripper at defcon 4 [03:38] i'm surprise this was defcon.org though [03:53] as part of one of the kickstarter campaigns I funded, they offered to make a game from an idea you had (very quick & dirty), so I put down saving web pages from the yahoo monster [04:34] 13â£4â£± 8â¡â£ 9â¢11â£ 12â£â¡ 13â 4â¡8â¢ 9â¢â¡ 11â¢¹12â 13â¢â¡ 4â¢8â£ 9â£â£ [04:34] 4â 8â ¸ 9â 11â £12â ¤ 13â â ¸ 4â 8â ±9â 11â £â 12â ¸ 4â £â 8â £9â ¼ 11â â 12â [04:35] that needs to scroll right-to-left [04:36] 13ï¼¡4ï½8ï½9ï½11ï½12ï½13ï½4ï¼´8ï½9ï½11ï½ [04:37] fabulous [04:46] chronomex: op me por favor <3 [04:50] Hey so I'm interested in helping out with archiveteam, is there anything I can do? I have some experience writing web scrapers (python) if that is needed. [04:52] for starters you can run the script at http://archiveteam.org/index.php?title=MobileMe [04:55] >_> [04:55] ok, how much space should I allocate for it? [04:56] I think around 10-20 gigs per profile is fine [04:57] seesaw.sh uploads and deletes after every successful download, so you shouldn't have a lot of accumulation [05:00] ok running :) [05:14] :D [05:17] maybe put that at the end or something :P [05:18] 4,5o12,1h9,6 10,4g5,11o11,13d11,6 5,11m9,1y7,13 4,3e2,6y8,1e12,10s [05:19] I demand a useful topic [05:22] yipdw: Set up UTF8 support, mmkay? [05:25] nitro2k01: I have UTF-8 support on [05:25] that does not make the topic content useful [05:25] well, it is now [05:25] or, put another way, being able to see "SketchCow for President" does not make the topic content useful [05:25] :P [05:27] :D [07:19] DFJustin, are you demanding a +c ? [07:20] am I who's it what [07:22] oh banning colour codes or something? [07:23] nah I don't care [07:24] tpb magnet link archive guy posted his script: http://pastebin.com/8RXXthXB [07:24] "I went trough all the torrents, but as a logged out user. This may be the reason why I saw only 1.5m of torrents. [07:24] I went trough the torrent description sites like [07:24] It took the computer about 2 days. I can post the script somewhere if you want, it is in perl, it is not complicated at all. [07:24] by adding 1 to the $i [07:24] http://thepiratebay.se/torrent/$i [07:25] " [07:25] for reference: from the comments of https://thepiratebay.se/torrent/7016365/The_whole_Pirate_Bay_magnet_archive [07:25] doing it again but logged in would be the first thing i'd think of [07:25] ugh perl [07:25] kennethre: yeah :( [07:25] readable and short enough to easily rewrite though [07:26] yeah [07:26] i wonder how fast i could distribute that [07:26] DFJustin, I thought those colours were a form of protest [07:27] run the whole thing in 10 minutes or something ;) [07:28] kennethre, I think you could download the entire mobileme in a few days :) [07:29] how much disk space do you have? :D [07:29] Nemo_bis: hah, ephemeral so depends on where i'm uploading to ;) [07:30] kennethre, how fast are you able to upload to batcave, usually? [07:30] or to archive.org [07:30] i don't have metrics [07:30] ah, I see underscor is downloading again [07:30] ok [07:30] yep [07:31] Got 2 fresh 2TB drives [07:31] :D [07:31] Plus another 500GB [07:31] i got 180 GB uploaded in <20 hours (iirc) [07:31] underscor, ah, so you're not using the IA machine? [07:31] Liability [07:32] oic [07:32] If it's downloaded from IA IP space, then there are more legal options [07:32] since IA is a "company"/library [07:32] also, they like to keep an arms-length from us [07:32] :D [07:32] since we're kinda crazy and maniacal [07:33] and uploading from the IA machines is not a problem? [07:33] what's being downloaded now? [07:33] mobileme [07:33] Nemo_bis: where's the script? [07:33] Nemo_bis: I dunno, I just want to play it safe [07:33] I don't have a lot of space on my VM anyway [07:34] kennethre, http://www.archiveteam.org/index.php?title=MobileMe [07:34] https://github.com/ArchiveTeam/mobileme-grab/ [07:34] underscor, add it to the topic [07:34] (the wiki page url) [07:34] aww, /this/ is useful topic now [07:36] sweet [07:37] i guess i can spin up a few vms :) [07:37] can i use the same upload slots i used for splinder? [07:37] No, there's a different one [07:37] see the upload-finished.sh script [07:38] i'll need a slot immediately [07:38] Use that, otherwise completion won't get reported to the tracker [07:38] You don't need a slot provisioned [07:38] excellent [07:38] so, we need a +77 % speed increase of the project to complete it [07:38] Really?? [07:39] well then [07:39] i'm on it :) [07:39] 200 instances should do [07:39] kennethre: heroku again? [07:39] :P [07:39] underscor, yes and it was way worse before you restarted [07:39] underscor: but of course : [07:39] seesaw supposedly uploads and deletes completed profiles while downloading or something [07:39] (I haven't looked at it) [07:39] kennethre: These are between a few k and 20GB, so hopefully that's not an issue [07:40] well i can't run a job for more than 24 hours [07:40] one in ...? [07:40] *once [07:41] so a few ginormous ones might need to be re-downloaded [07:41] You don't think heroku will boot you off? [07:41] underscor: i work for heroku [07:41] oh [07:41] :) [07:41] hahahahahahaha [07:42] hehe [07:42] niiice [07:42] how long has this been going on? [07:42] just noticed that the splinder readme had mobileme stuff in it [07:42] "Hey, uh, our bandwidth usage spiked up to 10 gigabits last night. You don't know anything about it, do you?" [07:43] Uh, long time ago [07:43] Before Halloween [07:43] But we didn't actively start until recently [07:43] cf taiwan floods [07:43] thailand floods? [07:44] er, whoops [07:44] thanks [07:45] yeah, seesaw.sh downloads one user, uploads it to batcave, deletes it locally, notifies the tracker, and starts over [07:45] oh geeze [07:45] that's perfect [07:45] hmm [07:46] https://github.com/kennethreitz/heroku-splinder/blob/master/run.py [07:46] so here's what i did before [07:46] https://github.com/kennethreitz/heroku-splinder/blob/master/Makefile [07:46] i'll prob keep using streamer [07:47] Why do that over seesaw? [07:47] different upload location, btw [07:47] (also it's called dld-client now) [07:48] i hope the one drive with that rsync module doesn't get overloaded [07:49] because seesaw only does one at a time, right? [07:49] no concurrency [07:49] so when would your system actually upload anything? [07:50] every 2 minutes it ran the upload script [07:50] in the background [07:50] none of the scripts do concurrency right now [07:50] dld-client nor seesaw [07:50] You just run multiples [07:50] dld-streamer.sh did ;( [07:50] A single thread will max out a 100mbit connection though [07:50] ah they're nice and fast compared to splinder then [07:50] It's akamai [07:51] splinder was all kinds of busted [07:51] shouldn't be an issue then :) [07:51] damn that's a ton of data [07:52] yeah [07:52] 211TB [07:52] estimated [07:52] :D [07:52] seesaw is awesome [07:52] that saves me so much trouble [07:52] anyone doing anything for imgur.com? lots of stuff uses that for hosting. [07:52] splinder was lots of tiny high-latency requests [07:53] yeah [07:53] i wasn't aware mobileme sites were on akamai [07:53] i know the real mobileme had a ton of issues [07:53] i wonder if the data i'm pulling down from my s3 bucket has been marked done [07:53] We should do an imgur mirror [07:54] https://www.youtube.com/watch?v=9MkbLQWHcXw [07:54] but they delete [07:54] :| [07:54] underscor: i bet there's a ton of illegal content [07:54] and i forget if i uploaded it to the batcave. (i suspect i did) [07:54] kennethre: Archive first, (don't) delete later ;) [07:54] haha [07:55] Yeah basically :> [07:56] wow and get-wget-warc too [07:56] this is way better [07:56] easy as pie [07:56] is there a verbose mode for any of this stuff? [07:56] i pretty much can run and look at logs [07:56] Does heroku give you basically a VM or something? [07:56] underscor: kinda, abstracted away [07:57] DYNOS! [07:57] :P [07:57] yipdw: :) [07:57] also yes there is a ton of illegal content on mobileme [07:57] I must have like gigs of new music [07:57] oh imeant imgur [07:57] most of which I am sure does not come with rights for redistribution [07:58] ohh [07:58] yeah that too :P [07:58] I must have like gigs of new music [07:58] hahaha [07:58] interesting project would be to pull out all the music we find [07:58] I haven't actually checked all the WARCs [07:58] or movies [07:58] but some of the biggest ones I've found contain either that or QuickTime movies [07:58] I'm sure there's lots of movies too [07:59] home movies most likely [07:59] so ./seesaw.sh just exits when a user is done [07:59] oh, it uploads the user's data, deletes it, goes for another [07:59] so autonomous, excellent [08:00] i assume it repeats and repeats [08:00] yeah [08:00] it's too bad MobileMe is not hosted on e.g. S3 [08:01] Why? [08:01] because then Heroku dynos would have a hilarious speed advantage [08:01] yeah [08:01] (afaik they're on EC2 instances) [08:01] and free [08:01] heh [08:01] oh, hahah [08:01] transfer [08:02] we'll get crazy speed anyway [08:02] i feel like upload will be the bottleneck [08:02] :P [08:02] I look forward to seeing kennethre bogart the dashboard [08:03] i really hope it happens :) [08:03] not fair >:| [08:03] I've been trying hard to get to the top spot [08:03] i'm not getting my hopes up [08:05] alright, here goes nothing. [08:05] how many processes? [08:05] haven't turned it on yet :) [08:05] oh [08:05] alright, 1 [08:05] ONE [08:06] gah failure [08:06] gah it relies on wget [08:10] That's what get-wget-warc does [08:10] Oh wait, does get-wget-warc call wget? [08:10] haha [08:10] yeah it does [08:10] i swapped it out with curl [08:10] alright i'm in action [08:11] i need to write an archive team buildpack [08:11] that has warc prebuilt [08:11] and rsync [08:11] the next wget release will have WARC writing capabilities present, but I'm not sure when that release will be [08:11] kennethre, yes, I doubt you'll be able to upload everything immediately [08:11] batcave is too slow for that [08:12] it's fast here [08:12] and it doesn't have a lot of space [08:12] I regularly get at least 1 MB/s [08:12] yipdw, how fast? [08:12] /dev/vdc 16T 15T 1.5T 91% /2 [08:12] uh oh [08:12] that's slow :-p [08:12] D: [08:12] it's not too bad [08:12] (that's the disk mobileme uploads are going to) [08:12] hmm [08:12] maybe those should be cleared off [08:12] it is if you download at some 10-100 MB/s as kennethre probably will :p [08:13] it's not too bad so far [08:13] he has so far downloaded < 1 MB [08:13] :P [08:13] haha [08:13] building now [08:13] underscor: where exactly is the mobileme data going to end up? [08:13] 200 TB is quite a tall order to provision [08:14] SketchCow is talking with archive folks [08:14] I think the current plan is to stick things in temporary items on the cluster [08:14] then slowly "download" them and reorganize and then reupload [08:14] Where's the dash? [08:15] Since they don't have 200TB of disks just laying around [08:15] hahaha [08:15] ersi: The dash? [08:15] +board [08:15] memac.heroku [08:15] .com [08:16] neat [08:18] If I'm running dld-client.sh, does stats get reported to the tracker? [08:18] yeah [08:19] alright [08:19] i think i'm in action [08:20] hroku scale scrape=100 [08:20] :D [08:21] do you get free resources? [08:21] yeah, my bill is $0 every month [08:21] no matter what i use [08:21] obviously i shouldn't do anything crazy [08:21] hopefully this won't be too intensive [08:23] TThat's cool [08:24] wow it's flying [08:24] this worries me [08:24] in the right on the dashboard [08:25] those are already-uploaded, right? [08:25] No, those are completed [08:26] They don't get reshown when they upload [08:26] Are you using seesaw? [08:26] i am [08:26] you weren't kidding when you said it'd exhaust the connection [08:26] damn [08:26] aaand there it goes [08:27] ho' shi', boy! [08:27] that's crazy [08:27] I want to see cacti/mrtg graphs for the ec2 instances [08:27] :D [08:27] hopefully this will help you keep up :) [08:27] this kind of saturation worries me [08:27] i need to check a few things [08:27] haha [08:27] i wasn't expecting it to go nearly this fast [08:28] "Uh, Kenneth. You broke EVERYTHING!" [08:28] nah not at all [08:28] haha [08:28] just abusing resources ;) [08:28] "slightly" [08:29] can someone tell me how much my user has uploaded so far? [08:29] is there an api for that? [08:29] just trying to gauge the current speed [08:29] kennethre: I can, one sec [08:29] No API, but I can du your module [08:30] it's been up for ~8 minutes [08:31] du: cannot read directory `kenneth/m': Permission denied [08:31] Oh, nevermind then [08:31] :( [08:31] I don't have sudo rights on SketchCow's box [08:32] kennethre: Upload your files with global read and exec [08:32] :P [08:32] Everyone elses are 755 [08:32] underscor: hey i'm just running the script :) [08:32] Yours are 700 :P [08:33] well a bit late for that now ;) [08:33] Heroku bug? [08:33] Well [08:33] Not bug [08:33] but default [08:33] prob just a configuration difference [08:33] yeah [08:34] Hmm [08:34] Oh, hey, mobileme isn't on /2 [08:34] Whoops [08:34] haha [08:34] /dev/vdd 11T 8.4T 2.2T 80% /3 [08:34] So, little more breathing room [08:34] if you're curious, here's what i'm doing https://github.com/kennethreitz/heroku-mobileme [08:35] Wow, you're killing the dashboard [08:37] :) [08:37] this is far more fun than you think. [08:37] i could scale it up way higher [08:37] but i think this'll do for now [08:38] underscor: probably another umask for his user, either on the upload box or download boxes [08:39] kennethre: /me waits for akamai to call up heroku [08:39] "Hey, uh, could you stop ddosing us?" [08:39] :D [08:39] that would deeply suck [08:39] i think they'd call ec2 first [08:39] I bet Apple will get a nice bill from Akamai [08:39] hehe [08:39] I'm sure that they're used to much higher load [08:39] Monetary Denial of Service [08:39] HAH [08:40] yeah, I bet they see hueg spikes occationally [08:40] have you seen their room? [08:40] Batcave is receiving at 10MB/s [08:40] but yeah, I bet kennethre is throwing them a punch haha [08:40] (so 80mbps) [08:40] http://www.youtube.com/watch?v=kiZgsBUEJdA [08:40] (Inside Akamai's war room) [08:41] i think it's pretty obvious we're not ddosing [08:41] Oh man [08:41] That's cool! [08:41] I want to visit that room [08:41] it's not like we're slow lorising them :) [08:41] hehe [08:41] oh man, youtube commercials for the fail [08:41] Oh, wow [08:42] It's Ahhh kamai [08:42] not aaaaa kamai [08:42] I've been saying it wrong for a while [08:42] start spelling it out instead, a k a m a i [08:43] can't go wrong with that [08:43] :D [08:44] Is anyone else pausing it so you can read everything on each display? [08:44] haha [08:44] http://www.youtube.com/watch?feature=player_detailpage&v=kiZgsBUEJdA#t=26s [08:44] 2.6 terabits/sec?!?!?!?! [08:45] hahaha [08:45] That's SO COOL [08:45] they power like 70% of the internet [08:45] haha [08:45] 12.7 million requests per second [08:45] wow. [08:46] kennethre: Batcave intake has jumped from ~50-80mbps to ~184-220mbps since you started [08:46] kennethre: https://github.com/kennethreitz/heroku-mobileme/commit/3f8ce58702cdbdf5dca3f48e4aa112b111d2a3a1 <-- hah what [08:46] yipdw: yeah, i'm just telling heroku it's a python app [08:47] yipdw: since i'm too lazy to do a custom buildpack [08:47] heh [08:47] underscor: that's freaking awesome ;) [08:47] haha [08:47] I wish I had sudo [08:47] underscor: what kind of connection does that thing have? [08:47] Then I could run iftop [08:48] Heh, first user I got; - Running wget --mirror (at least 16520 files)... [08:48] 400MB downloaded atm [08:48] kennethre: Gigabit copper [08:49] underscor: strait to a backbone? :P [08:49] No [08:49] i'm joking [08:49] I have a whole 1 mbps up [08:49]