#archiveteam 2011-11-22,Tue

โ†‘back Search

Time Nickname Message
00:29 ๐Ÿ”— pberry Noticed that the Splinder page mentioned that the project is "Closing"
00:30 ๐Ÿ”— dashcloud can someone tell me how close to the leaderboard I am?
00:35 ๐Ÿ”— pberry I just saw your name scroll by on the dashboard
00:36 ๐Ÿ”— pberry http://splinder.heroku.com/
02:49 ๐Ÿ”— bsmith094 now thats an interesting question http://www.archive.org/post/401968/we-have-2tb-of-data-to-upload is the data secure? depends, secure from the world seeing it, hell no, secure as in backed up safely, well probably (?)
02:53 ๐Ÿ”— SketchCow R-sync
03:05 ๐Ÿ”— dashcloud I get the feeling they think IA is a service they can subscribe to and use to hold files- otherwise, the questions make very little sense
03:07 ๐Ÿ”— SketchCow I'm asking some questions, then will likely upload Jamendo Albums.
03:14 ๐Ÿ”— STR_IDENT note to self: try not to crash znc, then forget passwords.
03:29 ๐Ÿ”— kennethre SketchCow: can I get an rsync slot?
03:30 ๐Ÿ”— kennethre mostly testing this system i'm doing right now to do this in massive parrallel on heroku (for free)
03:31 ๐Ÿ”— kennethre SketchCow: nick 'kennethreitz'
03:36 ๐Ÿ”— underscor wait
03:36 ๐Ÿ”— underscor THE kenneth reitz?
03:36 ๐Ÿ”— underscor readibility guy?
03:36 ๐Ÿ”— kennethre ..
03:36 ๐Ÿ”— underscor readability*
03:37 ๐Ÿ”— kennethre underscor: hah, the one and only :)
03:37 ๐Ÿ”— kennethre underscor: and you are?
03:37 ๐Ÿ”— underscor No way, rad!
03:37 ๐Ÿ”— underscor That's so cool!
03:37 ๐Ÿ”— kennethre how do you know of me? :P
03:37 ๐Ÿ”— underscor Readability and httpbin
03:38 ๐Ÿ”— kennethre ah, awesome :)
03:38 ๐Ÿ”— underscor :)
03:38 ๐Ÿ”— kennethre really want to start getting involved with what the archive team is doing
03:38 ๐Ÿ”— underscor That's great
03:38 ๐Ÿ”— kennethre i've always had a bit of an archival spirit
03:38 ๐Ÿ”— chronomex shocker :P
03:38 ๐Ÿ”— kennethre haha
03:40 ๐Ÿ”— kennethre going to try to run a few hundred of these scrapers in parallel on heroku
03:44 ๐Ÿ”— kennethre have it mostly working
03:44 ๐Ÿ”— kennethre a few annoyances (the image doesn't have rsync or wget)
04:01 ๐Ÿ”— SketchCow Hey, kennethre
04:01 ๐Ÿ”— SketchCow hahahahah
04:01 ๐Ÿ”— SketchCow underscor wants your autograph
04:02 ๐Ÿ”— kennethre SketchCow: apparently :)
04:02 ๐Ÿ”— SketchCow <3 <3 <3 your photo on the wall <3 <3 <3
04:02 ๐Ÿ”— SketchCow I'll get a slot for you shortly.
04:02 ๐Ÿ”— SketchCow Just finishing something here.
04:02 ๐Ÿ”— kennethre SketchCow: the one from the readability party?
04:02 ๐Ÿ”— kennethre excellent
04:07 ๐Ÿ”— SketchCow http://www.archive.org/details/moves-magazine
04:07 ๐Ÿ”— SketchCow Years ago, a guy named Greg Costikyan added all this description of all the issues of that magazine. I wrote him and he let me put them all up.
04:08 ๐Ÿ”— SketchCow So that's another thing off my plate.
04:08 ๐Ÿ”— underscor <SketchCow> <3 <3 <3 your photo on the wall <3 <3 <3
04:08 ๐Ÿ”— underscor :P
04:08 ๐Ÿ”— underscor You're just mad because I never asked for your autograph
04:08 ๐Ÿ”— SketchCow It's OK, I know mine's still in the place of honor on the back of the door
04:11 ๐Ÿ”— underscor :D
04:11 ๐Ÿ”— underscor I'm just trying to get in your will
04:11 ๐Ÿ”— underscor "And to Alex, I leave my trusty infocube, and well-loved collection of 1970s hustlers"
04:20 ๐Ÿ”— kennethre muahaha http://cl.ly/293x113C1z2F300m3Q36
04:23 ๐Ÿ”— underscor kennethre: Don't those cost money?
04:23 ๐Ÿ”— kennethre underscor: each 'process' (e.g. session) is $0.05 an hour
04:23 ๐Ÿ”— kennethre except the first one
04:24 ๐Ÿ”— kennethre so i can just run this on 50 apps
04:24 ๐Ÿ”— kennethre for free
04:25 ๐Ÿ”— kennethre just not sure how much space a dyno has available
04:26 ๐Ÿ”— NotGLaDOS I think the free ones have somewhere around 5MB or something
04:26 ๐Ÿ”— kennethre it's quite beautiful
04:26 ๐Ÿ”— Coderjoe ew
04:26 ๐Ÿ”— kennethre no
04:26 ๐Ÿ”— kennethre there's plenty of room
04:27 ๐Ÿ”— NotGLaDOS I have no idea how Heroku works though, so meh.
04:27 ๐Ÿ”— kennethre NotGLaDOS: i start working there in a few weeks
04:28 ๐Ÿ”— NotGLaDOS That helps.
04:28 ๐Ÿ”— kennethre and then i can use as much as i want for free
04:28 ๐Ÿ”— kennethre and plan to take advantage of that
04:28 ๐Ÿ”— NotGLaDOS Muahaha
04:28 ๐Ÿ”— kennethre :)
04:28 ๐Ÿ”— kennethre my friend works there, and his bill is ~10k a month
04:28 ๐Ÿ”— NotGLaDOS I should really use that free VPS someone randomly gave me.
04:39 ๐Ÿ”— kennethre muahaha completely free heroku app now
04:39 ๐Ÿ”— kennethre this is going to end well
04:40 ๐Ÿ”— kennethre heroku scale scrape=100
04:40 ๐Ÿ”— kennethre just need to get the upload triggering automatically somehow
04:40 ๐Ÿ”— DFJustin heh we're gonna run out of splinder at this rate
04:42 ๐Ÿ”— SketchCow Well, the QA is going to be (slightly) murder.
04:42 ๐Ÿ”— SketchCow We need to think that out
04:49 ๐Ÿ”— no2pencil I like murder
04:51 ๐Ÿ”— no2pencil & the xfiles marathon continues
04:51 ๐Ÿ”— no2pencil bitches
04:53 ๐Ÿ”— no2pencil oops
04:53 ๐Ÿ”— no2pencil :P
04:53 ๐Ÿ”— no2pencil wrong channel
04:53 ๐Ÿ”— no2pencil I was supposed to gloat elsewhere
04:58 ๐Ÿ”— SketchCow HERE WE GO
04:58 ๐Ÿ”— SketchCow Amiga World is on the way!
05:11 ๐Ÿ”— Coderjoe is there a channel for universal-tracker development discussion?
05:11 ๐Ÿ”— Coderjoe (whee. ruby. haven't touched that yet)
05:14 ๐Ÿ”— SketchCow #splinder has some chatting.
07:08 ๐Ÿ”— chronomex ^
07:21 ๐Ÿ”— SketchCow So, some time ago, someone sent me a hard drive for a geocities torrent.
07:21 ๐Ÿ”— SketchCow I was slow.
07:21 ๐Ÿ”— SketchCow Month or something.
07:21 ๐Ÿ”— SketchCow He switches from "where is it" to well, downright abusive.
07:21 ๐Ÿ”— SketchCow Well, there we have a problem.
07:22 ๐Ÿ”— SketchCow So I've had his hard drive on a shelf for, oh, 9 months now.
07:22 ๐Ÿ”— chronomex lol
07:22 ๐Ÿ”— SketchCow And every, oh, month or so, he sends me a more abusive, more threatening letter.
07:22 ๐Ÿ”— SketchCow I only mention because a few came in today.
07:22 ๐Ÿ”— SketchCow He also discovered my kickstarter.
07:22 ๐Ÿ”— SketchCow So now he's hoppin' mad this evening.
07:23 ๐Ÿ”— SketchCow I literally have the drive sitting on a shelf addressed to him.
07:23 ๐Ÿ”— SketchCow But I have a thing about threats and bullies.
07:23 ๐Ÿ”— Coderjoe speaking of hard drives, did you manage to get the friendster data out of my drive?
07:24 ๐Ÿ”— Coderjoe last I recall you were still working on finding a system to mount the xfs on
07:25 ๐Ÿ”— SketchCow It is just sitting there.
07:25 ๐Ÿ”— SketchCow But I'll try again.
07:25 ๐Ÿ”— SketchCow xfs is tough, man.
07:25 ๐Ÿ”— SketchCow It really is.
07:25 ๐Ÿ”— SketchCow P.s. try to avoid threatening my life in e-mail
07:25 ๐Ÿ”— SketchCow (Spoiler)
07:26 ๐Ÿ”— SketchCow I see now he is trying to derail my kickstarter.
07:26 ๐Ÿ”— SketchCow That'll be a trick.
07:26 ๐Ÿ”— chronomex especially since you got the money already
07:27 ๐Ÿ”— chronomex how's he trying?
07:27 ๐Ÿ”— SketchCow Well, I think he intends to donate $10 so he's a backer and can comment.
07:28 ๐Ÿ”— SketchCow (Doesn't work that way.)
07:28 ๐Ÿ”— SketchCow He did comment on there anyway, but I reported it as spam, so it's gone.
07:28 ๐Ÿ”— SketchCow I expect to have to talk/hang with a few kickstarter people.
07:29 ๐Ÿ”— chronomex well great, he gave you money.
07:30 ๐Ÿ”— chronomex kickstarter: turning misguided enemies into misinformed allies
07:30 ๐Ÿ”— Coderjoe woo
07:30 ๐Ÿ”— Coderjoe http://www.ncdc.noaa.gov/nexradinv/chooseday.jsp?id=kgrr
07:30 ๐Ÿ”— Coderjoe level 2 and level 3 radar data from 1995 to present
07:31 ๐Ÿ”— Coderjoe (that particular link goes to the grand rapids, mi radar site)
07:31 ๐Ÿ”— NotGLaDOS That's nice
07:31 ๐Ÿ”— NotGLaDOS Let me guess, id == location
07:32 ๐Ÿ”— Coderjoe it sounds like they need to retrieve it off tape in a robot, though
07:32 ๐Ÿ”— Coderjoe yes
07:32 ๐Ÿ”— SketchCow http://www.archive.org/details/amiga-world&reCache=1
07:32 ๐Ÿ”— chronomex Coderjoe: that is not a surprise.
07:32 ๐Ÿ”— NotGLaDOS Of course it does ( รฏยฝยก รฃยƒยฎรฏยพยŸ)
07:32 ๐Ÿ”— SketchCow All issues of Amiga World
07:32 ๐Ÿ”— Coderjoe The user email address is needed when ordering data from NCDC. Due to the size of the Archive, instantaneous access is not possible. The user is emailed when the ordered data has posted to the NCDC FTP site.
07:32 ๐Ÿ”— Coderjoe Why do I need to enter my Email Address?
07:32 ๐Ÿ”— Coderjoe How long will my order take?
07:32 ๐Ÿ”— Coderjoe The amount of time for each order is varies based on the size of the order. An average order of 24 hours of data may take between 5 and 30 minutes.
07:33 ๐Ÿ”— Coderjoe i hope there are multiple copies of the data
07:33 ๐Ÿ”— Coderjoe perhaps talk to them about loading it into ia? :D
07:33 ๐Ÿ”— NotGLaDOS So, basically, it'd be faster if we bought the NOAA out.
07:34 ๐Ÿ”— underscor How many stations are there?
07:34 ๐Ÿ”— chronomex maybe hundred or so, max
07:35 ๐Ÿ”— SketchCow OK, gotta go to bed again
07:35 ๐Ÿ”— SketchCow Hooray Alternate Side of the Street Parking
07:35 ๐Ÿ”— chronomex SketchCow has to go down
07:35 ๐Ÿ”— SketchCow Like your mom
07:35 ๐Ÿ”— chronomex like twitter or splinder, but he goes down IN BED
07:35 ๐Ÿ”— SketchCow LIKE YOUR MOM
07:36 ๐Ÿ”— chronomex NO
07:36 ๐Ÿ”— chronomex NOT THAT MENTAL IMAGE
07:36 ๐Ÿ”— underscor 16 years*365 days*100 sites*116 products per site
07:36 ๐Ÿ”— SketchCow Include the logs link
07:36 ๐Ÿ”— NotGLaDOS There goes the logs.
07:36 ๐Ÿ”— chronomex fine fine fine
07:37 ๐Ÿ”— underscor Roughly 67744000 requests to NCDC
07:37 ๐Ÿ”— underscor Do you think they'll mind?
07:37 ๐Ÿ”— SketchCow CHRONOMEX DESTROYS HISTORY
07:37 ๐Ÿ”— Coderjoe underscor: 365.25 days
07:37 ๐Ÿ”— chronomex underscor: probly.
07:37 ๐Ÿ”— chronomex SketchCow: jesus h christ
07:37 ๐Ÿ”— NotGLaDOS Probably not.
07:37 ๐Ÿ”— underscor 67790400
07:37 ๐Ÿ”— SketchCow Shut up, you will love austin
07:37 ๐Ÿ”— underscor There
07:37 ๐Ÿ”— NotGLaDOS It'll give them a reason to work 24/7
07:37 ๐Ÿ”— SketchCow we'll be such assholes at every panel
07:37 ๐Ÿ”— chronomex SketchCow: <3
07:37 ๐Ÿ”— Coderjoe underscor: I think you'd destroy their tape robot
07:37 ๐Ÿ”— SketchCow "WHERE'S THE EXPORT FUNCTION"
07:37 ๐Ÿ”— underscor I'm still mad I'm not going
07:37 ๐Ÿ”— underscor fucking money
07:37 ๐Ÿ”— SketchCow There are better things to be mad about
07:37 ๐Ÿ”— SketchCow that nad cyst
07:37 ๐Ÿ”— chronomex SketchCow: I'm thinking of calling all the startup hipster tards evil, unless they actively do good shit.
07:38 ๐Ÿ”— SketchCow Is that cleared up yet?
07:38 ๐Ÿ”— chronomex underscor: :|
07:38 ๐Ÿ”— underscor nah
07:38 ๐Ÿ”— underscor they want it to grow bigger first
07:38 ๐Ÿ”— SketchCow What's the theory
07:38 ๐Ÿ”— chronomex underscor: yeah, that nad cyst. lance that buboe
07:38 ๐Ÿ”— SketchCow the terrible, terrible theory
07:38 ๐Ÿ”— underscor It's just a benign back of baby juice
07:38 ๐Ÿ”— underscor backup*
07:38 ๐Ÿ”— chronomex baby juice or other fluids.
07:38 ๐Ÿ”— SketchCow THE SPERM DEATHSTAR
07:38 ๐Ÿ”— chronomex underscor: does this mean you need to jerk it more, or less?
07:39 ๐Ÿ”— Coderjoe need to relieve some pressure
07:39 ๐Ÿ”— underscor she said it shouldn't affect it
07:39 ๐Ÿ”— underscor lol
07:39 ๐Ÿ”— chronomex underscor: did you ask or did she volunteer it?
07:39 ๐Ÿ”— SketchCow "Should I keep up my usual three a day or do I have to ease back a bit"
07:39 ๐Ÿ”— chronomex *snerk*
07:39 ๐Ÿ”— underscor I was like "It's already the size of two fucking grapes, how big do you want it to fucking be?
07:40 ๐Ÿ”— chronomex underscor: four big mutant grapes, ime.
07:40 ๐Ÿ”— chronomex maybe three.
07:40 ๐Ÿ”— * SketchCow installs the Image::DoNotWant perl module
07:40 ๐Ÿ”— underscor hahahaha
07:40 ๐Ÿ”— Coderjoe chronomex: you have experience with nadcysts?
07:40 ๐Ÿ”— underscor Not that I'm gettin' it down with the ladies
07:40 ๐Ÿ”— SketchCow SOMEONE IS TRADING HEAVILY ON THE NADSAQ
07:40 ๐Ÿ”— underscor but it's totally obvious
07:41 ๐Ÿ”— underscor I don't even know how to fucking describe it
07:41 ๐Ÿ”— chronomex underscor: wait until it gets tender.
07:41 ๐Ÿ”— underscor It's like a baloon that uninflated unevenly
07:41 ๐Ÿ”— chronomex then be like HOLY FUCK FIX THIS SHIT NOW
07:41 ๐Ÿ”— underscor balloon*
07:41 ๐Ÿ”— bsmith093 so i can leave the streamer and the upload script running at once, right
07:41 ๐Ÿ”— underscor It's always tender
07:41 ๐Ÿ”— chronomex bsmith093: that's the design, yes.
07:41 ๐Ÿ”— chronomex SketchCow: what panel are you on?
07:41 ๐Ÿ”— underscor it's a tissue covered water balloon :V
07:42 ๐Ÿ”— SketchCow http://expertlabs.aaas.org/thinkup-launcher/
07:42 ๐Ÿ”— underscor that's rad
07:42 ๐Ÿ”— chronomex aaas, hm. I crashed their expo when it was in town few years ago.
07:43 ๐Ÿ”— chronomex I also jumped the fence at the association of american geographers expo last year
07:43 ๐Ÿ”— chronomex jumping the fence at an academic conference is a very surreal experience
07:43 ๐Ÿ”— chronomex (figurative fence)
07:44 ๐Ÿ”— bsmith093 ./upload-finished.sh batcave.textfiles.com::bsmith/splinder/
07:44 ๐Ÿ”— bsmith093 sending incremental file list after some minor hiccups, this is what i have for output, plus a crapload of filetranfers
07:46 ๐Ÿ”— NotGLaDOS Cameron_D is still pulling, according to top
07:46 ๐Ÿ”— Cameron_D Mhmm, Stuck on several large profiles methinks
07:46 ๐Ÿ”— Coderjoe dashboard says only 78k left
07:47 ๐Ÿ”— Coderjoe er, 72k
07:47 ๐Ÿ”— chronomex 72k for first run, doesn't hurt to do a second scan ;)
07:47 ๐Ÿ”— Coderjoe I know there are some unfinished IDs in there
07:47 ๐Ÿ”— chronomex lolz.
07:49 ๐Ÿ”— Coderjoe monsterfail
07:49 ๐Ÿ”— Coderjoe http://i.imgur.com/WZL4K.png
07:49 ๐Ÿ”— kennethre ..
07:49 ๐Ÿ”— NotGLaDOS I should really test the speed of my VPS..
07:50 ๐Ÿ”— NotGLaDOS Ah, 7M/s
07:50 ๐Ÿ”— bsmith093 is there any good reason the file structure is like this it/A/Ak/Aka/Akarui_Tenshi/
07:51 ๐Ÿ”— chronomex yes.
07:51 ๐Ÿ”— Coderjoe bsmith093: to keep the number of entries per directory down
07:51 ๐Ÿ”— bsmith093 are there any hard entry limits fs wise
07:51 ๐Ÿ”— chronomex no but if it gets big then your computer hates you.
07:51 ๐Ÿ”— chronomex depends on the filesystem, there's always a limit but it's usually over four billion
07:52 ๐Ÿ”— chronomex after a few thousand things start to get really slow
07:52 ๐Ÿ”— Coderjoe over ni..... beh. forget it
07:52 ๐Ÿ”— kennethre much easier to shard
07:52 ๐Ÿ”— kennethre or whatever you call that in archive land
07:52 ๐Ÿ”— chronomex yeah, whatever we call that
07:52 ๐Ÿ”— chronomex split?
07:52 ๐Ÿ”— chronomex sharding works fine.
07:52 ๐Ÿ”— kennethre Horizontal partitioning
07:53 ๐Ÿ”— bsmith093 chronomex: goo lord man, i dont have that many files (digitally) in my entire house, inclusing dupes
07:54 ๐Ÿ”— chronomex I probably have between 20 and 100 million files.
07:54 ๐Ÿ”— bsmith093 afk not have to sleep, its almost 3am inthe east coast, where i am
07:55 ๐Ÿ”— kennethre bsmith093: same here *sips coffee*
07:55 ๐Ÿ”— bsmith093 have too sleep now, typos becoming serious problem :O)
07:56 ๐Ÿ”— Coderjoe pardon me while I catalog another 1.5TB (after which the cataloged total will be 13.5TB >_< )
07:56 ๐Ÿ”— bsmith093 afk 4 ~10hrs
07:56 ๐Ÿ”— Coderjoe cataloging because currently finding a file on them is a pain in the arse
07:56 ๐Ÿ”— NotGLaDOS Coderjoe: thats a lot of data.
07:57 ๐Ÿ”— NotGLaDOS Wait, what'd happen if archive.org went down?
07:57 ๐Ÿ”— Coderjoe oh hay. this disk has toonami captures on it.
07:58 ๐Ÿ”— chronomex NotGLaDOS: it's unlikely to disappear, as there are multiple copies in multiple places.
07:58 ๐Ÿ”— NotGLaDOS True.
07:58 ๐Ÿ”— chronomex if it were to go away, though, history'd be up shit creek without much of a paddle
07:58 ๐Ÿ”— chronomex but I trust brewster to do the right thing.
08:01 ๐Ÿ”— chronomex well, mostly -- I never met him but I hear he's got his head screwed on right
08:02 ๐Ÿ”— Coderjoe damn how I miss circa-2000 cartoon network
08:45 ๐Ÿ”— chronomex I <3 my workplace
08:45 ๐Ÿ”— chronomex boss gave me and coworker lockpick sets today
08:45 ๐Ÿ”— chronomex "here's your thanksgiving bonus, use it to go get your christmas bonus"
08:52 ๐Ÿ”— yipdw haha
09:00 ๐Ÿ”— yipdw oh
09:03 ๐Ÿ”— yipdw splindid
09:03 ๐Ÿ”— yipdw splinder's back up
09:03 ๐Ÿ”— chronomex maybe we should try and not kick it offline.
09:03 ๐Ÿ”— yipdw yes
09:04 ๐Ÿ”— yipdw I'm just trying to finish up incompletes
09:05 ๐Ÿ”— yipdw huh, how do I add people to a github organization?
09:06 ๐Ÿ”— yipdw oh, you have to add them to a group, I see
09:10 ๐Ÿ”— yipdw hm.
09:10 ๐Ÿ”— yipdw is there a problem with just adding everyone we know to a "Contributors" team in the Github account
09:10 ๐Ÿ”— yipdw ?
09:11 ๐Ÿ”— yipdw with push/pull access to all repositories
09:19 ๐Ÿ”— yipdw huh
09:19 ๐Ÿ”— yipdw I think that, if anyone is retrieving US data right now
09:19 ๐Ÿ”— yipdw you should consider it suspect
09:20 ๐Ÿ”— yipdw I just finished up four US Splinder profiles in about two seconds, but they're all full of 404s
10:52 ๐Ÿ”— Coderjoe i'm still riding a wave of error 6 on both it and us profiles while waiting for the script to wind down
10:53 ๐Ÿ”— Cameron_D My script has been winding down for more than 12 hours
10:54 ๐Ÿ”— Cameron_D still 20 running, and quite a lot of error 6
11:19 ๐Ÿ”— Nemo_bis I'm fixing users with errors, but does the script find them all?
11:19 ๐Ÿ”— Nemo_bis For instance: http://toolserver.org/~nemobis/89gocciolina89-wget-phase-1.log has some 504 errors, but mostly "no data received" with no code
11:31 ๐Ÿ”— Nemo_bis ndurner, how many incomplete users do you have?
11:32 ๐Ÿ”— ndurner Nemo_bis: how do I know?
11:33 ๐Ÿ”— ndurner count ".incomplete", probably. Doing that now.
11:33 ๐Ÿ”— Nemo_bis ndurner, well, depends on your method; I look at the open dld-clients and at the numbers in dld-streamer
11:34 ๐Ÿ”— * Nemo_bis has 170
11:51 ๐Ÿ”— ndurner splinder.heroku.com: -6 to do
12:03 ๐Ÿ”— Cameron_D how do we go about verifiying if users are complete/fixing them?
12:06 ๐Ÿ”— ndurner upload_finished.sh just uploads completed ones, and if understood that correctly SketchCow is willing to run verification scripts on the data on batcave
12:11 ๐Ÿ”— Cameron_D Ah, I won't be able to upload until after the site has closed though
12:17 ๐Ÿ”— ndurner why not?
12:18 ๐Ÿ”— Cameron_D Nearly at my bandwidth cap
12:18 ๐Ÿ”— Cameron_D And at the rate I upload it will probably take a few days anyway
12:25 ๐Ÿ”— alard There are 266,205 users claimed but still not returned.
12:26 ๐Ÿ”— alard Time to add them back to the queue, I guess?
12:36 ๐Ÿ”— NotGLaDOS Sure, I suppose I could pull it out on my Romanian server.
12:36 ๐Ÿ”— NotGLaDOS (it'll probably fail, but still)
12:37 ๐Ÿ”— Cameron_D I tried on my cheap VPS, the disk IO killed it
13:06 ๐Ÿ”— Nemo_bis my fix-dld was "downloading" a bunch of users, but they were actually empty: wget-phase-1.log with error 404 and nothing else, only ~15 KiB downloaded
13:06 ๐Ÿ”— Nemo_bis hmmm http://www.us.splinder.com/
13:43 ๐Ÿ”— Coderjoe wow
13:43 ๐Ÿ”— Coderjoe www.splinder.com was very slow loading, but it finally did load
13:53 ๐Ÿ”— ndurner I have 953 .incompletes
13:54 ๐Ÿ”— Coderjoe mmm
13:54 ๐Ÿ”— Coderjoe 2011-11-22 11:50:03 ERROR 502: Bad Gateway.
13:55 ๐Ÿ”— Coderjoe the only error i see in this one "error 6" italian profile
14:01 ๐Ÿ”— bsmith093 plenty of error 6 too
14:01 ๐Ÿ”— ersi http://news.cnet.com/8301-17852_3-57329204-71/microsofts-new-incentive-for-engineering-hires-bacon/
14:02 ๐Ÿ”— ersi awesome
15:48 ๐Ÿ”— dnova why are a quarter of a million or so users gone from the total now>
15:49 ๐Ÿ”— pberry wakey wakey, eggs and ... bacon...ey
15:53 ๐Ÿ”— alard dnova: 259613 users have been claimed but never marked 'done'. total = done + todo + out, but the number out is not shown.
15:54 ๐Ÿ”— dnova oh
15:54 ๐Ÿ”— dnova well if the tracker is no longer providing names, should I touch STOP and let my 600 threads wind down naturally?
15:54 ๐Ÿ”— dnova or will you be adding more names back to the queue?
15:54 ๐Ÿ”— alard Earlier today I added 8000 users back to the todo queue, users that had been claimed for more than two days.
15:55 ๐Ÿ”— alard But the results I got back didn't look very healthy, since they were returned really quickly and were almost empty.
15:56 ๐Ÿ”— alard www.splinder.com is very slow at the moment and www.us.splinder.com is down, so maybe it's better to wait.
15:56 ๐Ÿ”— pberry I got a tiny bit of data before the tracker ran out of names, but still need to get a "rsync slot"
15:56 ๐Ÿ”— pberry only 29M
15:58 ๐Ÿ”— closure ah, out would be a good number to show
15:58 ๐Ÿ”— alard pberry: Can you make a tar file and upload it somewhere?
15:59 ๐Ÿ”— alard closure: Well, yes, on the other hand: out shouldn't be so enormously high.
16:00 ๐Ÿ”— closure well, it's ridiculous to think that 259 thousand grabs are currently running.. I'll bet something dropped those on the floor, either not done or done and the tracker not told
16:01 ๐Ÿ”— kennethre i was running 7k concurrently when the site went down yesterday
16:05 ๐Ÿ”— pberry alard: you bet
16:05 ๐Ÿ”— pberry alard: like, just the data directory?
16:05 ๐Ÿ”— alard It makes me wonder: would it help to run so many at the same time?
16:05 ๐Ÿ”— alard pberry: Yes.
16:06 ๐Ÿ”— pberry alard: as soon as these last few threads stop I'll get right on that
16:15 ๐Ÿ”— dnova alard: I have around 5,200 incompletes and around 520 threads still running
16:15 ๐Ÿ”— dnova roughly
16:15 ๐Ÿ”— dnova the last ones are probably bigger profiles
16:26 ๐Ÿ”— pberry I love the ones that are 4 files and take forever
16:31 ๐Ÿ”— Coderjoe closure: or errord. or dropped on the floor yesterday when clients were forcibly killed when splinder went down
16:32 ๐Ÿ”— Coderjoe (yesterday morning, eastern US time)
16:33 ๐Ÿ”— Coderjoe my ec2 instance has 2598 .incomplete files
16:33 ๐Ÿ”— Coderjoe and currently 131 at home
17:05 ๐Ÿ”— SketchCow HI
19:02 ๐Ÿ”— bsmith093 just got back did my rsync complete, because the tracker for the streamer is apparently down
19:08 ๐Ÿ”— SketchCow We downloaded it
19:08 ๐Ÿ”— SketchCow Now we need to do a cleanup phase
19:09 ๐Ÿ”— bsmith093 all of it?
19:14 ๐Ÿ”— SketchCow Yes.
19:14 ๐Ÿ”— SketchCow We had downtimes in there.
19:14 ๐Ÿ”— SketchCow It won't be hard, I'm sure it'll be a script that generates a new list.
19:15 ๐Ÿ”— bsmith093 im running the fix script now, i have a really long list of apparently incomplete profiles
19:16 ๐Ÿ”— bsmith093 so you thought you'd be 1.5 days too late, and you're 2 days early?
19:17 ๐Ÿ”— SketchCow We had someone come in with 300 virtual machines
19:18 ๐Ÿ”— bsmith093 how does that work, dont they all need some significant space to actually store the data
19:19 ๐Ÿ”— kennethre webscale
19:19 ๐Ÿ”— kennethre er, "the cloud"
19:20 ๐Ÿ”— bsmith093 oh heroku, then?
19:20 ๐Ÿ”— kennethre yessir
19:20 ๐Ÿ”— bsmith093 so how much of their space did u buy/ rent?
19:22 ๐Ÿ”— Coderjoe this really didn't need much space
19:22 ๐Ÿ”— bsmith093 i saw the dashborad, 400gb for blogs alone
19:22 ๐Ÿ”— kennethre every 'dyno' (e.g. intance) is a self-contained virtualized environment
19:22 ๐Ÿ”— kennethre *instance
19:22 ๐Ÿ”— bsmith093 but it dtill need hd space in some form
19:23 ๐Ÿ”— kennethre the cedar stack has writable storage
19:23 ๐Ÿ”— Coderjoe bsmith093: that's all 1 million users (or so). each user is typically 1MB or less
19:23 ๐Ÿ”— kennethre but it dies when the dyno dies (when the process stops)
19:23 ๐Ÿ”— Coderjoe (yes there were some that got into double digits)
19:23 ๐Ÿ”— bsmith093 1mil at only 400gb, wow im impressed at how much people didnt care about this blog thing?
19:24 ๐Ÿ”— kennethre so the 'process' just ran the stream downloader, and uploaded ever 200 seconds
19:24 ๐Ÿ”— kennethre across 300 "boxes"
19:24 ๐Ÿ”— Coderjoe kennethre: sounds like ec2 instance-store... but your resource problems make it sound like the dyno isn't running on a bare instance
19:24 ๐Ÿ”— kennethre my spelling is wonderful today
19:24 ๐Ÿ”— kennethre Coderjoe: it's not, they use LXC
19:24 ๐Ÿ”— bsmith093 but where did the writable storage ultimately dump to?
19:25 ๐Ÿ”— kennethre bsmith093: it disappears when the process dies
19:25 ๐Ÿ”— kennethre it's all gone now
19:25 ๐Ÿ”— Coderjoe which explains your resource issues when you were forking too much
19:25 ๐Ÿ”— kennethre that's why i was continually uploading
19:25 ๐Ÿ”— closure it's encoded in a laser beam being bounced off the moon for now, we'll put it somewhere later
19:25 ๐Ÿ”— kennethre Coderjoe: yeah, at 20 i didn't have a problem, 60 i did
19:25 ๐Ÿ”— bsmith093 unless im missing something that means the data goes too, so where did... oh ok then
19:25 ๐Ÿ”— kennethre bsmith093: rsync
19:25 ๐Ÿ”— Coderjoe mmm.. delay line memory
19:26 ๐Ÿ”— bsmith093 lol laser
19:26 ๐Ÿ”— bsmith093 yeah that makes much more sense
19:26 ๐Ÿ”— kennethre it would stay as long as the processes stay alive, but they get recycled every day i believe
19:26 ๐Ÿ”— kennethre and that gets pricy when there's 300 of them ;)
19:26 ๐Ÿ”— bsmith093 sent 128293146 bytes received 42113 bytes 100222.77 bytes/sec total size is 4697371387 speedup is 36.60 rsync error: some files/attrs were not transferred (see previous errors) (code 23) at main.c(1060) [sender=3.0.7]
19:26 ๐Ÿ”— bsmith093 \is that error serious?
19:27 ๐Ÿ”— bsmith093 thats my re run of the rsync that i left running all night
19:28 ๐Ÿ”— Coderjoe re-run. it should tell you what the error was, and shouldn't upload anything new (as long as everything got uploaded already)
19:35 ๐Ÿ”— bsmith093 ben@ben-laptop:~/splinder-grab$ ./upload-finished.sh batcave.textfiles.com::bsmith/splinder/ sending incremental file list rsync: link_stat "/home/ben/splinder-grab/data/it/g/gy/gyp/gypsy!" failed: No such file or directory (2)
19:35 ๐Ÿ”— bsmith093 sent 1841728 bytes received 12173 bytes 50791.81 bytes/sec
19:37 ๐Ÿ”— bsmith093 so anyway, I just checked the projects page, good for you, whoever's saving ff.net and fictionpress, but may i suggest a linklist of straight story urls, with this app, works great for me, fanficdownloader.net
19:38 ๐Ÿ”— Coderjoe mmm
19:38 ๐Ÿ”— bsmith093 simple text files with all sotry urls in it one to a line, goes through aves in whatever format you want, even plain text formatted _like_ *this* for bold and italic and underlined things
19:38 ๐Ÿ”— Coderjoe I think rsync (or more likely the upload-finished.sh script) is choking on that !
19:39 ๐Ÿ”— bsmith093 it might be the ! char
19:39 ๐Ÿ”— Coderjoe er, no that is rsync spewing the error
19:40 ๐Ÿ”— bsmith093 still thats one profile out of hundreds of thousands, whose gonna notice?
19:42 ๐Ÿ”— Coderjoe oh it's just the one site on geocities that had recordings and stuff of radio transmissions from jonestown... who's gonna notice?
19:43 ๐Ÿ”— bsmith093 all right allright point taken
19:43 ๐Ÿ”— bsmith093 incidentally are there audio recording =s from jonestownn?
19:44 ๐Ÿ”— Coderjoe yes.
19:44 ๐Ÿ”— Coderjoe I think you can get them if you file a FOIA request to someone
19:44 ๐Ÿ”— bsmith093 huh imagine that, the balls o those people recording a cult leader
19:44 ๐Ÿ”— bsmith093 oh well foia
19:45 ๐Ÿ”— bsmith093 anyway can we ping the archive yet and see if someone else has theat profile?
19:45 ๐Ÿ”— Coderjoe these were recordings made at a monitoring station of radio transmissions peoples church members were making
19:45 ๐Ÿ”— bsmith093 wow they suspected something, that strongly?
19:46 ๐Ÿ”— Coderjoe as far as I have been able to tell, none of the geocities archive projects managed to get that profile
19:46 ๐Ÿ”— bsmith093 oh you were serious about that, damn that sucks
19:46 ๐Ÿ”— Schbirid those recordings are at archive.org
19:46 ๐Ÿ”— Schbirid just went through the internets a couiple of days ago
19:46 ๐Ÿ”— Schbirid probably on metafilter, check there
19:46 ๐Ÿ”— Coderjoe http://boingboing.net/2008/11/19/jonestown-30-years-l-2.html
19:47 ๐Ÿ”— Coderjoe Schbirid: oh? cool
19:47 ๐Ÿ”— Schbirid http://www.archive.org/details/ptc1978-11-18.flac16
19:53 ๐Ÿ”— bsmith093 how much do we still need to get from mobilme
19:53 ๐Ÿ”— bsmith093 data in gb
20:04 ๐Ÿ”— underscor <chronomex> "here's your thanksgiving bonus, use it to go get your christmas bonus"
20:04 ๐Ÿ”— underscor <chronomex> I <3 my workplace
20:04 ๐Ÿ”— underscor <chronomex> boss gave me and coworker lockpick sets today
20:04 ๐Ÿ”— underscor haha
20:09 ๐Ÿ”— kennethre underscor: hah, awesome. Where do you work?
20:10 ๐Ÿ”— Schbirid he could tell you, but ... !
20:10 ๐Ÿ”— underscor Oh, I do work at the archive in return for... well. In return for the knowledge that stuff will be saved
20:10 ๐Ÿ”— underscor That's not my story, it's chronomex's
20:10 ๐Ÿ”— underscor (the lockpick thing)
20:52 ๐Ÿ”— bsmith093 are there any slightly smaller projects than mobilme i could help out with?
21:01 ๐Ÿ”— PatC How could I join the 'Archive Team'?
21:07 ๐Ÿ”— bsmith093 how do i put wget warc in its usr bin place so i can call it normally
21:10 ๐Ÿ”— underscor sudo cp ./wget-warc /usr/bin
21:10 ๐Ÿ”— underscor hash -r
21:10 ๐Ÿ”— db48x I would do mkdir ~/bin; cp ./wget-warc ~/bin
21:10 ๐Ÿ”— db48x then add ~/bin to my PATH
21:10 ๐Ÿ”— db48x export PATH=$PATH:~/bin
21:11 ๐Ÿ”— bsmith093 i ust cped to usr bin, that seems to work thansk
21:11 ๐Ÿ”— bsmith093 ok i apparently meant how do i compile it with a]man entries and all that?
21:12 ๐Ÿ”— bsmith093 to make updateable and everythikng
21:12 ๐Ÿ”— db48x change the get-wget-warc.sh script so that it doesn't delete the source directory after it builds it
21:12 ๐Ÿ”— db48x then go in there and do sudo make install
21:13 ๐Ÿ”— db48x this won't build a deb or rpm package, so your package manager won't be able to keep it up to date
21:45 ๐Ÿ”— anonymous I need an rsync slot
21:49 ๐Ÿ”— underscor ...
22:30 ๐Ÿ”— closure he's on 4chan, just post the slot there
22:30 ๐Ÿ”— Coderjoe he went over to #splinder
22:46 ๐Ÿ”— dashcloud so I should stop all splinder downloads now?
23:51 ๐Ÿ”— Paradoks underscor: Why does ATidlebot think you're offline?
23:54 ๐Ÿ”— db48x2 closure: hah
23:58 ๐Ÿ”— Paradoks PatC: I don't know if you got an answer elsewhere, but as far as joining Archive Team, well, see the #archiveteam title. As far as what could you do, right now, well, we're working on downloading me.com stuff.
23:58 ๐Ÿ”— Paradoks See http://www.archiveteam.org/index.php?title=MobileMe for details.
23:58 ๐Ÿ”— Paradoks ...including how to run a BASH script on your linux box.
23:59 ๐Ÿ”— PatC 10-4
23:59 ๐Ÿ”— PatC thanks

irclogger-viewer