#archiveteam 2011-11-22,Tue

↑back Search

Time	Nickname	Message
00:29 ^🔗	pberry	Noticed that the Splinder page mentioned that the project is "Closing"
00:30 ^🔗	dashcloud	can someone tell me how close to the leaderboard I am?
00:35 ^🔗	pberry	I just saw your name scroll by on the dashboard
00:36 ^🔗	pberry	http://splinder.heroku.com/
02:49 ^🔗	bsmith094	now thats an interesting question http://www.archive.org/post/401968/we-have-2tb-of-data-to-upload is the data secure? depends, secure from the world seeing it, hell no, secure as in backed up safely, well probably (?)
02:53 ^🔗	SketchCow	R-sync
03:05 ^🔗	dashcloud	I get the feeling they think IA is a service they can subscribe to and use to hold files- otherwise, the questions make very little sense
03:07 ^🔗	SketchCow	I'm asking some questions, then will likely upload Jamendo Albums.
03:14 ^🔗	STR_IDENT	note to self: try not to crash znc, then forget passwords.
03:29 ^🔗	kennethre	SketchCow: can I get an rsync slot?
03:30 ^🔗	kennethre	mostly testing this system i'm doing right now to do this in massive parrallel on heroku (for free)
03:31 ^🔗	kennethre	SketchCow: nick 'kennethreitz'
03:36 ^🔗	underscor	wait
03:36 ^🔗	underscor	THE kenneth reitz?
03:36 ^🔗	underscor	readibility guy?
03:36 ^🔗	kennethre	..
03:36 ^🔗	underscor	readability*
03:37 ^🔗	kennethre	underscor: hah, the one and only :)
03:37 ^🔗	kennethre	underscor: and you are?
03:37 ^🔗	underscor	No way, rad!
03:37 ^🔗	underscor	That's so cool!
03:37 ^🔗	kennethre	how do you know of me? :P
03:37 ^🔗	underscor	Readability and httpbin
03:38 ^🔗	kennethre	ah, awesome :)
03:38 ^🔗	underscor	:)
03:38 ^🔗	kennethre	really want to start getting involved with what the archive team is doing
03:38 ^🔗	underscor	That's great
03:38 ^🔗	kennethre	i've always had a bit of an archival spirit
03:38 ^🔗	chronomex	shocker :P
03:38 ^🔗	kennethre	haha
03:40 ^🔗	kennethre	going to try to run a few hundred of these scrapers in parallel on heroku
03:44 ^🔗	kennethre	have it mostly working
03:44 ^🔗	kennethre	a few annoyances (the image doesn't have rsync or wget)
04:01 ^🔗	SketchCow	Hey, kennethre
04:01 ^🔗	SketchCow	hahahahah
04:01 ^🔗	SketchCow	underscor wants your autograph
04:02 ^🔗	kennethre	SketchCow: apparently :)
04:02 ^🔗	SketchCow	<3 <3 <3 your photo on the wall <3 <3 <3
04:02 ^🔗	SketchCow	I'll get a slot for you shortly.
04:02 ^🔗	SketchCow	Just finishing something here.
04:02 ^🔗	kennethre	SketchCow: the one from the readability party?
04:02 ^🔗	kennethre	excellent
04:07 ^🔗	SketchCow	http://www.archive.org/details/moves-magazine
04:07 ^🔗	SketchCow	Years ago, a guy named Greg Costikyan added all this description of all the issues of that magazine. I wrote him and he let me put them all up.
04:08 ^🔗	SketchCow	So that's another thing off my plate.
04:08 ^🔗	underscor	<SketchCow> <3 <3 <3 your photo on the wall <3 <3 <3
04:08 ^🔗	underscor	:P
04:08 ^🔗	underscor	You're just mad because I never asked for your autograph
04:08 ^🔗	SketchCow	It's OK, I know mine's still in the place of honor on the back of the door
04:11 ^🔗	underscor	:D
04:11 ^🔗	underscor	I'm just trying to get in your will
04:11 ^🔗	underscor	"And to Alex, I leave my trusty infocube, and well-loved collection of 1970s hustlers"
04:20 ^🔗	kennethre	muahaha http://cl.ly/293x113C1z2F300m3Q36
04:23 ^🔗	underscor	kennethre: Don't those cost money?
04:23 ^🔗	kennethre	underscor: each 'process' (e.g. session) is $0.05 an hour
04:23 ^🔗	kennethre	except the first one
04:24 ^🔗	kennethre	so i can just run this on 50 apps
04:24 ^🔗	kennethre	for free
04:25 ^🔗	kennethre	just not sure how much space a dyno has available
04:26 ^🔗	NotGLaDOS	I think the free ones have somewhere around 5MB or something
04:26 ^🔗	kennethre	it's quite beautiful
04:26 ^🔗	Coderjoe	ew
04:26 ^🔗	kennethre	no
04:26 ^🔗	kennethre	there's plenty of room
04:27 ^🔗	NotGLaDOS	I have no idea how Heroku works though, so meh.
04:27 ^🔗	kennethre	NotGLaDOS: i start working there in a few weeks
04:28 ^🔗	NotGLaDOS	That helps.
04:28 ^🔗	kennethre	and then i can use as much as i want for free
04:28 ^🔗	kennethre	and plan to take advantage of that
04:28 ^🔗	NotGLaDOS	Muahaha
04:28 ^🔗	kennethre	:)
04:28 ^🔗	kennethre	my friend works there, and his bill is ~10k a month
04:28 ^🔗	NotGLaDOS	I should really use that free VPS someone randomly gave me.
04:39 ^🔗	kennethre	muahaha completely free heroku app now
04:39 ^🔗	kennethre	this is going to end well
04:40 ^🔗	kennethre	heroku scale scrape=100
04:40 ^🔗	kennethre	just need to get the upload triggering automatically somehow
04:40 ^🔗	DFJustin	heh we're gonna run out of splinder at this rate
04:42 ^🔗	SketchCow	Well, the QA is going to be (slightly) murder.
04:42 ^🔗	SketchCow	We need to think that out
04:49 ^🔗	no2pencil	I like murder
04:51 ^🔗	no2pencil	& the xfiles marathon continues
04:51 ^🔗	no2pencil	bitches
04:53 ^🔗	no2pencil	oops
04:53 ^🔗	no2pencil	:P
04:53 ^🔗	no2pencil	wrong channel
04:53 ^🔗	no2pencil	I was supposed to gloat elsewhere
04:58 ^🔗	SketchCow	HERE WE GO
04:58 ^🔗	SketchCow	Amiga World is on the way!
05:11 ^🔗	Coderjoe	is there a channel for universal-tracker development discussion?
05:11 ^🔗	Coderjoe	(whee. ruby. haven't touched that yet)
05:14 ^🔗	SketchCow	#splinder has some chatting.
07:08 ^🔗	chronomex	^
07:21 ^🔗	SketchCow	So, some time ago, someone sent me a hard drive for a geocities torrent.
07:21 ^🔗	SketchCow	I was slow.
07:21 ^🔗	SketchCow	Month or something.
07:21 ^🔗	SketchCow	He switches from "where is it" to well, downright abusive.
07:21 ^🔗	SketchCow	Well, there we have a problem.
07:22 ^🔗	SketchCow	So I've had his hard drive on a shelf for, oh, 9 months now.
07:22 ^🔗	chronomex	lol
07:22 ^🔗	SketchCow	And every, oh, month or so, he sends me a more abusive, more threatening letter.
07:22 ^🔗	SketchCow	I only mention because a few came in today.
07:22 ^🔗	SketchCow	He also discovered my kickstarter.
07:22 ^🔗	SketchCow	So now he's hoppin' mad this evening.
07:23 ^🔗	SketchCow	I literally have the drive sitting on a shelf addressed to him.
07:23 ^🔗	SketchCow	But I have a thing about threats and bullies.
07:23 ^🔗	Coderjoe	speaking of hard drives, did you manage to get the friendster data out of my drive?
07:24 ^🔗	Coderjoe	last I recall you were still working on finding a system to mount the xfs on
07:25 ^🔗	SketchCow	It is just sitting there.
07:25 ^🔗	SketchCow	But I'll try again.
07:25 ^🔗	SketchCow	xfs is tough, man.
07:25 ^🔗	SketchCow	It really is.
07:25 ^🔗	SketchCow	P.s. try to avoid threatening my life in e-mail
07:25 ^🔗	SketchCow	(Spoiler)
07:26 ^🔗	SketchCow	I see now he is trying to derail my kickstarter.
07:26 ^🔗	SketchCow	That'll be a trick.
07:26 ^🔗	chronomex	especially since you got the money already
07:27 ^🔗	chronomex	how's he trying?
07:27 ^🔗	SketchCow	Well, I think he intends to donate $10 so he's a backer and can comment.
07:28 ^🔗	SketchCow	(Doesn't work that way.)
07:28 ^🔗	SketchCow	He did comment on there anyway, but I reported it as spam, so it's gone.
07:28 ^🔗	SketchCow	I expect to have to talk/hang with a few kickstarter people.
07:29 ^🔗	chronomex	well great, he gave you money.
07:30 ^🔗	chronomex	kickstarter: turning misguided enemies into misinformed allies
07:30 ^🔗	Coderjoe	woo
07:30 ^🔗	Coderjoe	http://www.ncdc.noaa.gov/nexradinv/chooseday.jsp?id=kgrr
07:30 ^🔗	Coderjoe	level 2 and level 3 radar data from 1995 to present
07:31 ^🔗	Coderjoe	(that particular link goes to the grand rapids, mi radar site)
07:31 ^🔗	NotGLaDOS	That's nice
07:31 ^🔗	NotGLaDOS	Let me guess, id == location
07:32 ^🔗	Coderjoe	it sounds like they need to retrieve it off tape in a robot, though
07:32 ^🔗	Coderjoe	yes
07:32 ^🔗	SketchCow	http://www.archive.org/details/amiga-world&reCache=1
07:32 ^🔗	chronomex	Coderjoe: that is not a surprise.
07:32 ^🔗	NotGLaDOS	Of course it does ( ï½¡ ã®ï¾)
07:32 ^🔗	SketchCow	All issues of Amiga World
07:32 ^🔗	Coderjoe	The user email address is needed when ordering data from NCDC. Due to the size of the Archive, instantaneous access is not possible. The user is emailed when the ordered data has posted to the NCDC FTP site.
07:32 ^🔗	Coderjoe	Why do I need to enter my Email Address?
07:32 ^🔗	Coderjoe	How long will my order take?
07:32 ^🔗	Coderjoe	The amount of time for each order is varies based on the size of the order. An average order of 24 hours of data may take between 5 and 30 minutes.
07:33 ^🔗	Coderjoe	i hope there are multiple copies of the data
07:33 ^🔗	Coderjoe	perhaps talk to them about loading it into ia? :D
07:33 ^🔗	NotGLaDOS	So, basically, it'd be faster if we bought the NOAA out.
07:34 ^🔗	underscor	How many stations are there?
07:34 ^🔗	chronomex	maybe hundred or so, max
07:35 ^🔗	SketchCow	OK, gotta go to bed again
07:35 ^🔗	SketchCow	Hooray Alternate Side of the Street Parking
07:35 ^🔗	chronomex	SketchCow has to go down
07:35 ^🔗	SketchCow	Like your mom
07:35 ^🔗	chronomex	like twitter or splinder, but he goes down IN BED
07:35 ^🔗	SketchCow	LIKE YOUR MOM
07:36 ^🔗	chronomex	NO
07:36 ^🔗	chronomex	NOT THAT MENTAL IMAGE
07:36 ^🔗	underscor	16 years365 days100 sites*116 products per site
07:36 ^🔗	SketchCow	Include the logs link
07:36 ^🔗	NotGLaDOS	There goes the logs.
07:36 ^🔗	chronomex	fine fine fine
07:37 ^🔗	underscor	Roughly 67744000 requests to NCDC
07:37 ^🔗	underscor	Do you think they'll mind?
07:37 ^🔗	SketchCow	CHRONOMEX DESTROYS HISTORY
07:37 ^🔗	Coderjoe	underscor: 365.25 days
07:37 ^🔗	chronomex	underscor: probly.
07:37 ^🔗	chronomex	SketchCow: jesus h christ
07:37 ^🔗	NotGLaDOS	Probably not.
07:37 ^🔗	underscor	67790400
07:37 ^🔗	SketchCow	Shut up, you will love austin
07:37 ^🔗	underscor	There
07:37 ^🔗	NotGLaDOS	It'll give them a reason to work 24/7
07:37 ^🔗	SketchCow	we'll be such assholes at every panel
07:37 ^🔗	chronomex	SketchCow: <3
07:37 ^🔗	Coderjoe	underscor: I think you'd destroy their tape robot
07:37 ^🔗	SketchCow	"WHERE'S THE EXPORT FUNCTION"
07:37 ^🔗	underscor	I'm still mad I'm not going
07:37 ^🔗	underscor	fucking money
07:37 ^🔗	SketchCow	There are better things to be mad about
07:37 ^🔗	SketchCow	that nad cyst
07:37 ^🔗	chronomex	SketchCow: I'm thinking of calling all the startup hipster tards evil, unless they actively do good shit.
07:38 ^🔗	SketchCow	Is that cleared up yet?
07:38 ^🔗	chronomex	underscor: :\|
07:38 ^🔗	underscor	nah
07:38 ^🔗	underscor	they want it to grow bigger first
07:38 ^🔗	SketchCow	What's the theory
07:38 ^🔗	chronomex	underscor: yeah, that nad cyst. lance that buboe
07:38 ^🔗	SketchCow	the terrible, terrible theory
07:38 ^🔗	underscor	It's just a benign back of baby juice
07:38 ^🔗	underscor	backup*
07:38 ^🔗	chronomex	baby juice or other fluids.
07:38 ^🔗	SketchCow	THE SPERM DEATHSTAR
07:38 ^🔗	chronomex	underscor: does this mean you need to jerk it more, or less?
07:39 ^🔗	Coderjoe	need to relieve some pressure
07:39 ^🔗	underscor	she said it shouldn't affect it
07:39 ^🔗	underscor	lol
07:39 ^🔗	chronomex	underscor: did you ask or did she volunteer it?
07:39 ^🔗	SketchCow	"Should I keep up my usual three a day or do I have to ease back a bit"
07:39 ^🔗	chronomex	snerk
07:39 ^🔗	underscor	I was like "It's already the size of two fucking grapes, how big do you want it to fucking be?
07:40 ^🔗	chronomex	underscor: four big mutant grapes, ime.
07:40 ^🔗	chronomex	maybe three.
07:40 ^🔗	*	SketchCow installs the Image::DoNotWant perl module
07:40 ^🔗	underscor	hahahaha
07:40 ^🔗	Coderjoe	chronomex: you have experience with nadcysts?
07:40 ^🔗	underscor	Not that I'm gettin' it down with the ladies
07:40 ^🔗	SketchCow	SOMEONE IS TRADING HEAVILY ON THE NADSAQ
07:40 ^🔗	underscor	but it's totally obvious
07:41 ^🔗	underscor	I don't even know how to fucking describe it
07:41 ^🔗	chronomex	underscor: wait until it gets tender.
07:41 ^🔗	underscor	It's like a baloon that uninflated unevenly
07:41 ^🔗	chronomex	then be like HOLY FUCK FIX THIS SHIT NOW
07:41 ^🔗	underscor	balloon*
07:41 ^🔗	bsmith093	so i can leave the streamer and the upload script running at once, right
07:41 ^🔗	underscor	It's always tender
07:41 ^🔗	chronomex	bsmith093: that's the design, yes.
07:41 ^🔗	chronomex	SketchCow: what panel are you on?
07:41 ^🔗	underscor	it's a tissue covered water balloon :V
07:42 ^🔗	SketchCow	http://expertlabs.aaas.org/thinkup-launcher/
07:42 ^🔗	underscor	that's rad
07:42 ^🔗	chronomex	aaas, hm. I crashed their expo when it was in town few years ago.
07:43 ^🔗	chronomex	I also jumped the fence at the association of american geographers expo last year
07:43 ^🔗	chronomex	jumping the fence at an academic conference is a very surreal experience
07:43 ^🔗	chronomex	(figurative fence)
07:44 ^🔗	bsmith093	./upload-finished.sh batcave.textfiles.com::bsmith/splinder/
07:44 ^🔗	bsmith093	sending incremental file list after some minor hiccups, this is what i have for output, plus a crapload of filetranfers
07:46 ^🔗	NotGLaDOS	Cameron_D is still pulling, according to top
07:46 ^🔗	Cameron_D	Mhmm, Stuck on several large profiles methinks
07:46 ^🔗	Coderjoe	dashboard says only 78k left
07:47 ^🔗	Coderjoe	er, 72k
07:47 ^🔗	chronomex	72k for first run, doesn't hurt to do a second scan ;)
07:47 ^🔗	Coderjoe	I know there are some unfinished IDs in there
07:47 ^🔗	chronomex	lolz.
07:49 ^🔗	Coderjoe	monsterfail
07:49 ^🔗	Coderjoe	http://i.imgur.com/WZL4K.png
07:49 ^🔗	kennethre	..
07:49 ^🔗	NotGLaDOS	I should really test the speed of my VPS..
07:50 ^🔗	NotGLaDOS	Ah, 7M/s
07:50 ^🔗	bsmith093	is there any good reason the file structure is like this it/A/Ak/Aka/Akarui_Tenshi/
07:51 ^🔗	chronomex	yes.
07:51 ^🔗	Coderjoe	bsmith093: to keep the number of entries per directory down
07:51 ^🔗	bsmith093	are there any hard entry limits fs wise
07:51 ^🔗	chronomex	no but if it gets big then your computer hates you.
07:51 ^🔗	chronomex	depends on the filesystem, there's always a limit but it's usually over four billion
07:52 ^🔗	chronomex	after a few thousand things start to get really slow
07:52 ^🔗	Coderjoe	over ni..... beh. forget it
07:52 ^🔗	kennethre	much easier to shard
07:52 ^🔗	kennethre	or whatever you call that in archive land
07:52 ^🔗	chronomex	yeah, whatever we call that
07:52 ^🔗	chronomex	split?
07:52 ^🔗	chronomex	sharding works fine.
07:52 ^🔗	kennethre	Horizontal partitioning
07:53 ^🔗	bsmith093	chronomex: goo lord man, i dont have that many files (digitally) in my entire house, inclusing dupes
07:54 ^🔗	chronomex	I probably have between 20 and 100 million files.
07:54 ^🔗	bsmith093	afk not have to sleep, its almost 3am inthe east coast, where i am
07:55 ^🔗	kennethre	bsmith093: same here sips coffee
07:55 ^🔗	bsmith093	have too sleep now, typos becoming serious problem :O)
07:56 ^🔗	Coderjoe	pardon me while I catalog another 1.5TB (after which the cataloged total will be 13.5TB >_< )
07:56 ^🔗	bsmith093	afk 4 ~10hrs
07:56 ^🔗	Coderjoe	cataloging because currently finding a file on them is a pain in the arse
07:56 ^🔗	NotGLaDOS	Coderjoe: thats a lot of data.
07:57 ^🔗	NotGLaDOS	Wait, what'd happen if archive.org went down?
07:57 ^🔗	Coderjoe	oh hay. this disk has toonami captures on it.
07:58 ^🔗	chronomex	NotGLaDOS: it's unlikely to disappear, as there are multiple copies in multiple places.
07:58 ^🔗	NotGLaDOS	True.
07:58 ^🔗	chronomex	if it were to go away, though, history'd be up shit creek without much of a paddle
07:58 ^🔗	chronomex	but I trust brewster to do the right thing.
08:01 ^🔗	chronomex	well, mostly -- I never met him but I hear he's got his head screwed on right
08:02 ^🔗	Coderjoe	damn how I miss circa-2000 cartoon network
08:45 ^🔗	chronomex	I <3 my workplace
08:45 ^🔗	chronomex	boss gave me and coworker lockpick sets today
08:45 ^🔗	chronomex	"here's your thanksgiving bonus, use it to go get your christmas bonus"
08:52 ^🔗	yipdw	haha
09:00 ^🔗	yipdw	oh
09:03 ^🔗	yipdw	splindid
09:03 ^🔗	yipdw	splinder's back up
09:03 ^🔗	chronomex	maybe we should try and not kick it offline.
09:03 ^🔗	yipdw	yes
09:04 ^🔗	yipdw	I'm just trying to finish up incompletes
09:05 ^🔗	yipdw	huh, how do I add people to a github organization?
09:06 ^🔗	yipdw	oh, you have to add them to a group, I see
09:10 ^🔗	yipdw	hm.
09:10 ^🔗	yipdw	is there a problem with just adding everyone we know to a "Contributors" team in the Github account
09:10 ^🔗	yipdw	?
09:11 ^🔗	yipdw	with push/pull access to all repositories
09:19 ^🔗	yipdw	huh
09:19 ^🔗	yipdw	I think that, if anyone is retrieving US data right now
09:19 ^🔗	yipdw	you should consider it suspect
09:20 ^🔗	yipdw	I just finished up four US Splinder profiles in about two seconds, but they're all full of 404s
10:52 ^🔗	Coderjoe	i'm still riding a wave of error 6 on both it and us profiles while waiting for the script to wind down
10:53 ^🔗	Cameron_D	My script has been winding down for more than 12 hours
10:54 ^🔗	Cameron_D	still 20 running, and quite a lot of error 6
11:19 ^🔗	Nemo_bis	I'm fixing users with errors, but does the script find them all?
11:19 ^🔗	Nemo_bis	For instance: http://toolserver.org/~nemobis/89gocciolina89-wget-phase-1.log has some 504 errors, but mostly "no data received" with no code
11:31 ^🔗	Nemo_bis	ndurner, how many incomplete users do you have?
11:32 ^🔗	ndurner	Nemo_bis: how do I know?
11:33 ^🔗	ndurner	count ".incomplete", probably. Doing that now.
11:33 ^🔗	Nemo_bis	ndurner, well, depends on your method; I look at the open dld-clients and at the numbers in dld-streamer
11:34 ^🔗	*	Nemo_bis has 170
11:51 ^🔗	ndurner	splinder.heroku.com: -6 to do
12:03 ^🔗	Cameron_D	how do we go about verifiying if users are complete/fixing them?
12:06 ^🔗	ndurner	upload_finished.sh just uploads completed ones, and if understood that correctly SketchCow is willing to run verification scripts on the data on batcave
12:11 ^🔗	Cameron_D	Ah, I won't be able to upload until after the site has closed though
12:17 ^🔗	ndurner	why not?
12:18 ^🔗	Cameron_D	Nearly at my bandwidth cap
12:18 ^🔗	Cameron_D	And at the rate I upload it will probably take a few days anyway
12:25 ^🔗	alard	There are 266,205 users claimed but still not returned.
12:26 ^🔗	alard	Time to add them back to the queue, I guess?
12:36 ^🔗	NotGLaDOS	Sure, I suppose I could pull it out on my Romanian server.
12:36 ^🔗	NotGLaDOS	(it'll probably fail, but still)
12:37 ^🔗	Cameron_D	I tried on my cheap VPS, the disk IO killed it
13:06 ^🔗	Nemo_bis	my fix-dld was "downloading" a bunch of users, but they were actually empty: wget-phase-1.log with error 404 and nothing else, only ~15 KiB downloaded
13:06 ^🔗	Nemo_bis	hmmm http://www.us.splinder.com/
13:43 ^🔗	Coderjoe	wow
13:43 ^🔗	Coderjoe	www.splinder.com was very slow loading, but it finally did load
13:53 ^🔗	ndurner	I have 953 .incompletes
13:54 ^🔗	Coderjoe	mmm
13:54 ^🔗	Coderjoe	2011-11-22 11:50:03 ERROR 502: Bad Gateway.
13:55 ^🔗	Coderjoe	the only error i see in this one "error 6" italian profile
14:01 ^🔗	bsmith093	plenty of error 6 too
14:01 ^🔗	ersi	http://news.cnet.com/8301-17852_3-57329204-71/microsofts-new-incentive-for-engineering-hires-bacon/
14:02 ^🔗	ersi	awesome
15:48 ^🔗	dnova	why are a quarter of a million or so users gone from the total now>
15:49 ^🔗	pberry	wakey wakey, eggs and ... bacon...ey
15:53 ^🔗	alard	dnova: 259613 users have been claimed but never marked 'done'. total = done + todo + out, but the number out is not shown.
15:54 ^🔗	dnova	oh
15:54 ^🔗	dnova	well if the tracker is no longer providing names, should I touch STOP and let my 600 threads wind down naturally?
15:54 ^🔗	dnova	or will you be adding more names back to the queue?
15:54 ^🔗	alard	Earlier today I added 8000 users back to the todo queue, users that had been claimed for more than two days.
15:55 ^🔗	alard	But the results I got back didn't look very healthy, since they were returned really quickly and were almost empty.
15:56 ^🔗	alard	www.splinder.com is very slow at the moment and www.us.splinder.com is down, so maybe it's better to wait.
15:56 ^🔗	pberry	I got a tiny bit of data before the tracker ran out of names, but still need to get a "rsync slot"
15:56 ^🔗	pberry	only 29M
15:58 ^🔗	closure	ah, out would be a good number to show
15:58 ^🔗	alard	pberry: Can you make a tar file and upload it somewhere?
15:59 ^🔗	alard	closure: Well, yes, on the other hand: out shouldn't be so enormously high.
16:00 ^🔗	closure	well, it's ridiculous to think that 259 thousand grabs are currently running.. I'll bet something dropped those on the floor, either not done or done and the tracker not told
16:01 ^🔗	kennethre	i was running 7k concurrently when the site went down yesterday
16:05 ^🔗	pberry	alard: you bet
16:05 ^🔗	pberry	alard: like, just the data directory?
16:05 ^🔗	alard	It makes me wonder: would it help to run so many at the same time?
16:05 ^🔗	alard	pberry: Yes.
16:06 ^🔗	pberry	alard: as soon as these last few threads stop I'll get right on that
16:15 ^🔗	dnova	alard: I have around 5,200 incompletes and around 520 threads still running
16:15 ^🔗	dnova	roughly
16:15 ^🔗	dnova	the last ones are probably bigger profiles
16:26 ^🔗	pberry	I love the ones that are 4 files and take forever
16:31 ^🔗	Coderjoe	closure: or errord. or dropped on the floor yesterday when clients were forcibly killed when splinder went down
16:32 ^🔗	Coderjoe	(yesterday morning, eastern US time)
16:33 ^🔗	Coderjoe	my ec2 instance has 2598 .incomplete files
16:33 ^🔗	Coderjoe	and currently 131 at home
17:05 ^🔗	SketchCow	HI
19:02 ^🔗	bsmith093	just got back did my rsync complete, because the tracker for the streamer is apparently down
19:08 ^🔗	SketchCow	We downloaded it
19:08 ^🔗	SketchCow	Now we need to do a cleanup phase
19:09 ^🔗	bsmith093	all of it?
19:14 ^🔗	SketchCow	Yes.
19:14 ^🔗	SketchCow	We had downtimes in there.
19:14 ^🔗	SketchCow	It won't be hard, I'm sure it'll be a script that generates a new list.
19:15 ^🔗	bsmith093	im running the fix script now, i have a really long list of apparently incomplete profiles
19:16 ^🔗	bsmith093	so you thought you'd be 1.5 days too late, and you're 2 days early?
19:17 ^🔗	SketchCow	We had someone come in with 300 virtual machines
19:18 ^🔗	bsmith093	how does that work, dont they all need some significant space to actually store the data
19:19 ^🔗	kennethre	webscale
19:19 ^🔗	kennethre	er, "the cloud"
19:20 ^🔗	bsmith093	oh heroku, then?
19:20 ^🔗	kennethre	yessir
19:20 ^🔗	bsmith093	so how much of their space did u buy/ rent?
19:22 ^🔗	Coderjoe	this really didn't need much space
19:22 ^🔗	bsmith093	i saw the dashborad, 400gb for blogs alone
19:22 ^🔗	kennethre	every 'dyno' (e.g. intance) is a self-contained virtualized environment
19:22 ^🔗	kennethre	*instance
19:22 ^🔗	bsmith093	but it dtill need hd space in some form
19:23 ^🔗	kennethre	the cedar stack has writable storage
19:23 ^🔗	Coderjoe	bsmith093: that's all 1 million users (or so). each user is typically 1MB or less
19:23 ^🔗	kennethre	but it dies when the dyno dies (when the process stops)
19:23 ^🔗	Coderjoe	(yes there were some that got into double digits)
19:23 ^🔗	bsmith093	1mil at only 400gb, wow im impressed at how much people didnt care about this blog thing?
19:24 ^🔗	kennethre	so the 'process' just ran the stream downloader, and uploaded ever 200 seconds
19:24 ^🔗	kennethre	across 300 "boxes"
19:24 ^🔗	Coderjoe	kennethre: sounds like ec2 instance-store... but your resource problems make it sound like the dyno isn't running on a bare instance
19:24 ^🔗	kennethre	my spelling is wonderful today
19:24 ^🔗	kennethre	Coderjoe: it's not, they use LXC
19:24 ^🔗	bsmith093	but where did the writable storage ultimately dump to?
19:25 ^🔗	kennethre	bsmith093: it disappears when the process dies
19:25 ^🔗	kennethre	it's all gone now
19:25 ^🔗	Coderjoe	which explains your resource issues when you were forking too much
19:25 ^🔗	kennethre	that's why i was continually uploading
19:25 ^🔗	closure	it's encoded in a laser beam being bounced off the moon for now, we'll put it somewhere later
19:25 ^🔗	kennethre	Coderjoe: yeah, at 20 i didn't have a problem, 60 i did
19:25 ^🔗	bsmith093	unless im missing something that means the data goes too, so where did... oh ok then
19:25 ^🔗	kennethre	bsmith093: rsync
19:25 ^🔗	Coderjoe	mmm.. delay line memory
19:26 ^🔗	bsmith093	lol laser
19:26 ^🔗	bsmith093	yeah that makes much more sense
19:26 ^🔗	kennethre	it would stay as long as the processes stay alive, but they get recycled every day i believe
19:26 ^🔗	kennethre	and that gets pricy when there's 300 of them ;)
19:26 ^🔗	bsmith093	sent 128293146 bytes received 42113 bytes 100222.77 bytes/sec total size is 4697371387 speedup is 36.60 rsync error: some files/attrs were not transferred (see previous errors) (code 23) at main.c(1060) [sender=3.0.7]
19:26 ^🔗	bsmith093	\is that error serious?
19:27 ^🔗	bsmith093	thats my re run of the rsync that i left running all night
19:28 ^🔗	Coderjoe	re-run. it should tell you what the error was, and shouldn't upload anything new (as long as everything got uploaded already)
19:35 ^🔗	bsmith093	ben@ben-laptop:~/splinder-grab$ ./upload-finished.sh batcave.textfiles.com::bsmith/splinder/ sending incremental file list rsync: link_stat "/home/ben/splinder-grab/data/it/g/gy/gyp/gypsy!" failed: No such file or directory (2)
19:35 ^🔗	bsmith093	sent 1841728 bytes received 12173 bytes 50791.81 bytes/sec
19:37 ^🔗	bsmith093	so anyway, I just checked the projects page, good for you, whoever's saving ff.net and fictionpress, but may i suggest a linklist of straight story urls, with this app, works great for me, fanficdownloader.net
19:38 ^🔗	Coderjoe	mmm
19:38 ^🔗	bsmith093	simple text files with all sotry urls in it one to a line, goes through aves in whatever format you want, even plain text formatted _like_ this for bold and italic and underlined things
19:38 ^🔗	Coderjoe	I think rsync (or more likely the upload-finished.sh script) is choking on that !
19:39 ^🔗	bsmith093	it might be the ! char
19:39 ^🔗	Coderjoe	er, no that is rsync spewing the error
19:40 ^🔗	bsmith093	still thats one profile out of hundreds of thousands, whose gonna notice?
19:42 ^🔗	Coderjoe	oh it's just the one site on geocities that had recordings and stuff of radio transmissions from jonestown... who's gonna notice?
19:43 ^🔗	bsmith093	all right allright point taken
19:43 ^🔗	bsmith093	incidentally are there audio recording =s from jonestownn?
19:44 ^🔗	Coderjoe	yes.
19:44 ^🔗	Coderjoe	I think you can get them if you file a FOIA request to someone
19:44 ^🔗	bsmith093	huh imagine that, the balls o those people recording a cult leader
19:44 ^🔗	bsmith093	oh well foia
19:45 ^🔗	bsmith093	anyway can we ping the archive yet and see if someone else has theat profile?
19:45 ^🔗	Coderjoe	these were recordings made at a monitoring station of radio transmissions peoples church members were making
19:45 ^🔗	bsmith093	wow they suspected something, that strongly?
19:46 ^🔗	Coderjoe	as far as I have been able to tell, none of the geocities archive projects managed to get that profile
19:46 ^🔗	bsmith093	oh you were serious about that, damn that sucks
19:46 ^🔗	Schbirid	those recordings are at archive.org
19:46 ^🔗	Schbirid	just went through the internets a couiple of days ago
19:46 ^🔗	Schbirid	probably on metafilter, check there
19:46 ^🔗	Coderjoe	http://boingboing.net/2008/11/19/jonestown-30-years-l-2.html
19:47 ^🔗	Coderjoe	Schbirid: oh? cool
19:47 ^🔗	Schbirid	http://www.archive.org/details/ptc1978-11-18.flac16
19:53 ^🔗	bsmith093	how much do we still need to get from mobilme
19:53 ^🔗	bsmith093	data in gb
20:04 ^🔗	underscor	<chronomex> "here's your thanksgiving bonus, use it to go get your christmas bonus"
20:04 ^🔗	underscor	<chronomex> I <3 my workplace
20:04 ^🔗	underscor	<chronomex> boss gave me and coworker lockpick sets today
20:04 ^🔗	underscor	haha
20:09 ^🔗	kennethre	underscor: hah, awesome. Where do you work?
20:10 ^🔗	Schbirid	he could tell you, but ... !
20:10 ^🔗	underscor	Oh, I do work at the archive in return for... well. In return for the knowledge that stuff will be saved
20:10 ^🔗	underscor	That's not my story, it's chronomex's
20:10 ^🔗	underscor	(the lockpick thing)
20:52 ^🔗	bsmith093	are there any slightly smaller projects than mobilme i could help out with?
21:01 ^🔗	PatC	How could I join the 'Archive Team'?
21:07 ^🔗	bsmith093	how do i put wget warc in its usr bin place so i can call it normally
21:10 ^🔗	underscor	sudo cp ./wget-warc /usr/bin
21:10 ^🔗	underscor	hash -r
21:10 ^🔗	db48x	I would do mkdir ~/bin; cp ./wget-warc ~/bin
21:10 ^🔗	db48x	then add ~/bin to my PATH
21:10 ^🔗	db48x	export PATH=$PATH:~/bin
21:11 ^🔗	bsmith093	i ust cped to usr bin, that seems to work thansk
21:11 ^🔗	bsmith093	ok i apparently meant how do i compile it with a]man entries and all that?
21:12 ^🔗	bsmith093	to make updateable and everythikng
21:12 ^🔗	db48x	change the get-wget-warc.sh script so that it doesn't delete the source directory after it builds it
21:12 ^🔗	db48x	then go in there and do sudo make install
21:13 ^🔗	db48x	this won't build a deb or rpm package, so your package manager won't be able to keep it up to date
21:45 ^🔗	anonymous	I need an rsync slot
21:49 ^🔗	underscor	...
22:30 ^🔗	closure	he's on 4chan, just post the slot there
22:30 ^🔗	Coderjoe	he went over to #splinder
22:46 ^🔗	dashcloud	so I should stop all splinder downloads now?
23:51 ^🔗	Paradoks	underscor: Why does ATidlebot think you're offline?
23:54 ^🔗	db48x2	closure: hah
23:58 ^🔗	Paradoks	PatC: I don't know if you got an answer elsewhere, but as far as joining Archive Team, well, see the #archiveteam title. As far as what could you do, right now, well, we're working on downloading me.com stuff.
23:58 ^🔗	Paradoks	See http://www.archiveteam.org/index.php?title=MobileMe for details.
23:58 ^🔗	Paradoks	...including how to run a BASH script on your linux box.
23:59 ^🔗	PatC	10-4
23:59 ^🔗	PatC	thanks

irclogger-viewer