#archiveteam 2011-11-16,Wed

↑back Search

Time Nickname Message
00:05 🔗 SketchCow Where's my hug
00:05 🔗 * marceloan hugs SketchCow
00:05 🔗 marceloan LOL
00:44 🔗 underscor likwid_: Willing to share that script?
00:44 🔗 closure ndurner1: wow, you're burning up the wires
00:47 🔗 RedType for a long time i wanted to record links people posted in irc and couldnt get a mysql database on the machine irssi ran on so i wrote a script which tweeted the links
00:47 🔗 RedType literally the first perl script i wrote after "hello world"
00:57 🔗 likwid_ underscor: sure, not done yet tho http://pastebin.com/JFgMKKJc
00:57 🔗 likwid_ underscor: right now it only gets 200 entries, but it does owrk
00:58 🔗 SketchCow THIS PLACE NEEDS MORE FIRE
00:58 🔗 underscor I went down down down in a burning ring of fire
00:58 🔗 likwid_ underscor: so I need to build in "pagination" kinda. Basically I have to refactor line 244 down into a function that I can call a bunch of times until results returned > 200
00:59 🔗 underscor oic, cool
00:59 🔗 RedType likwid_: are you using twitter's search api
00:59 🔗 underscor What goes in config.yaml?
00:59 🔗 likwid_ underscor: hopefully i'll have it done tonight, but not sure since I just remembered that I need to back up my LJ, hahahaha
00:59 🔗 underscor haha
00:59 🔗 likwid_ underscor: it should build that for you
00:59 🔗 underscor Well, gimme a poke when it's done, I'm definitely interested
00:59 🔗 underscor likwid_: Oh, great!
01:00 🔗 likwid_ underscor: cool :) yeah check out the create_default_config_file method
01:00 🔗 likwid_ the twitter boilerplate crap is from another thing I made
01:00 🔗 likwid_ but it works
01:00 🔗 underscor are these consumer key and secret tied to your account?
01:00 🔗 likwid_ underscor: they're tied to my developer account, yeah
01:01 🔗 likwid_ every app that uses the Twitter API needs an API key
01:01 🔗 underscor I meant are they okay for me to use, or should I sub them out?
01:01 🔗 likwid_ they're fine
01:01 🔗 underscor k, awesome
01:01 🔗 likwid_ just bound to a registered application that I own (twitter-avatar-update)...I'll probably make new ones before I release this
01:01 🔗 underscor I see :)
01:02 🔗 likwid_ damn I've had back luck with these Western Digital Green drives
01:02 🔗 likwid_ we have a stack of dead ones at work and my 2TB just died (WD20EARS)
01:03 🔗 likwid_ they last ~7 months
01:31 🔗 marceloan At the speed we're downloading AnyHub, when do you think will we finish?
01:34 🔗 underscor Well, we've done 28k in what, 3 days?
01:34 🔗 underscor so roughly 10k a day
01:34 🔗 underscor probably sometime tomorrow afternoon
02:03 🔗 underscor Hey, is anyhub down?
02:03 🔗 underscor Everything on the recently added page is empty
02:05 🔗 marceloan Should I touch stop?
02:05 🔗 underscor Dunno, maybe those are legitamately empty
02:06 🔗 chronomex don't touch that dial!
02:06 🔗 underscor Better to run them for now
02:06 🔗 marceloan I touched the STOP
02:40 🔗 marceloan Good Night
04:36 🔗 Hydria Hi, is SketchCow here?
04:36 🔗 underscor Might be
04:37 🔗 underscor This is the best time to catch him usually
04:37 🔗 underscor (late at night, not this exact time specifically)
04:37 🔗 Hydriz Man
04:38 🔗 Hydriz Did I miss him?
04:40 🔗 underscor http://games.yahoo.com/blogs/plugged-in/peta-slams-mario-over-fur-suit-211025773.html
04:40 🔗 underscor Nah, he flits in and out
04:41 🔗 Hydriz lol
04:42 🔗 * Hydriz needs a rsync slot
04:50 🔗 closure http://lefty333boy.splinder.com/ this is 4.4 gb, and counting
04:50 🔗 closure christ
04:51 🔗 underscor haha
04:52 🔗 underscor Gah, so many projects :(
04:52 🔗 underscor I wish they were more spaced out
05:00 🔗 Hydriz oh, I missed SketchCow 4 hours ago...
07:53 🔗 Hydriz Hello, is SketchCow online now?
07:54 🔗 db48x Hydriz: I don't think so
07:55 🔗 Hydriz ...
07:55 🔗 db48x Hydriz: why do you ask?
07:56 🔗 Hydriz I need an rsync module
07:56 🔗 db48x ah
07:56 🔗 Hydriz *slot
07:56 🔗 db48x send him a private message, he may see it before he hits the sack
07:57 🔗 chronomex maybe, it's like 3am where he lives.
07:57 🔗 chronomex though idfk when he sleeps
07:58 🔗 Hydriz SketchCow!
07:58 🔗 SketchCow Hey!!!!!!!!!
07:58 🔗 Hydriz I need an rsync slot to upload AnyHub grab
07:58 🔗 chronomex clearly he doesn't sleep right now
07:59 🔗 Hydriz lol
08:00 🔗 Hydriz I have about 60+ GB to transfer
08:01 🔗 Hydriz are you able to provide one?
08:01 🔗 SketchCow Yes
08:02 🔗 SketchCow Give me a moment, the IRC client a-sploded
08:02 🔗 Hydriz okie :)
08:04 🔗 Hydriz erm, do I have a time limit?
08:05 🔗 SketchCow Can you see the sun outside?
08:05 🔗 Hydriz yes
08:05 🔗 SketchCow OK.
08:05 🔗 SketchCow If that burns out, stop uploading.
08:05 🔗 ersi Ah, service.
08:05 🔗 chronomex hahaha
08:05 🔗 Hydriz though it is cloudy now haha
08:05 🔗 ersi RUN!
08:06 🔗 db48x ** TODO [#A] upload continuously
08:06 🔗 db48x DEADLINE: <5000000000-01-01>
08:07 🔗 SketchCow So, here's a project.
08:08 🔗 SketchCow A non-profit group just made a global social downloader.
08:08 🔗 ersi Deadline: 4,294,967,295
08:08 🔗 SketchCow Most of the guys in the group, I don't personally like.
08:08 🔗 SketchCow But the main coder? Love her. LOVE HER.
08:08 🔗 SketchCow So she made this thing.
08:08 🔗 SketchCow It's called ThinkUp
08:08 🔗 * db48x googles
08:08 🔗 SketchCow It's basically a personal social network backup
08:08 🔗 SketchCow Run it, it constantly keeps your shit backed up off these sites.
08:08 🔗 ersi http://thinkupapp.com/ ?
08:08 🔗 SketchCow Sounds right
08:09 🔗 SketchCow If it's got Anil Dash bloviating, that's it
08:09 🔗 ersi Looks jazzyshebang
08:09 🔗 SketchCow Yeah
08:09 🔗 SketchCow Then it drops you right into a fucking github
08:10 🔗 * ersi laughs
08:10 🔗 SketchCow Regardless
08:10 🔗 SketchCow it's a good idea
08:10 🔗 SketchCow I'd love for us to look at it
08:10 🔗 SketchCow It's open source
08:10 🔗 Cameron_D That is cool
08:10 🔗 SketchCow if it's really good, we should grab it and work with it
08:10 🔗 ersi Ah, gina's the main coder
08:10 🔗 SketchCow LOVE HER
08:10 🔗 ersi bumped into her on teh netz before
08:11 🔗 ersi ah, yeah - she writes for Lifehacker as well :-)
08:11 🔗 Hydriz SketchCow: Where is that rsync server located?
08:11 🔗 ersi In sunny california, where the disks - always go hot!
08:11 🔗 db48x 280 forks
08:12 🔗 Hydriz California...
08:12 🔗 ersi Yeah, that's where the Internet Archive lives
08:13 🔗 Hydriz ok, then afterI rsync I can delete everything I have?
08:13 🔗 ersi rsync a few times first
08:13 🔗 Hydriz nevermind, that was a dumb question
08:13 🔗 ersi just to make sure you got everything
08:13 🔗 Hydriz ok
08:13 🔗 ersi if you don't need the space immediately, it's always Very nice to have an extra copy laying around
08:13 🔗 chronomex ^
08:13 🔗 Hydriz just worried that we can't archive everything in time
08:13 🔗 Hydriz it is closing in two days
08:14 🔗 ersi I mean, mistakes happens.. things disappear and stuff
08:14 🔗 db48x I think we'll get anyhub
08:14 🔗 chronomex we request you keep everything around unless you run out of space
08:14 🔗 Hydriz I know
08:14 🔗 chronomex :)
08:14 🔗 Hydriz but this is not my server
08:14 🔗 chronomex aye
08:14 🔗 Cameron_D I'm throwing all my bandwidth at it, 300gb until I'm at my download limit
08:14 🔗 alard Hydriz: For uploading anyhub, please use the upload script from github. It'll upload things in a more or less standard way, which is handy for postprocessing.
08:14 🔗 Hydriz I ran 20 terminals for it :(
08:14 🔗 Hydriz yeap
08:14 🔗 Hydriz following the script
08:14 🔗 alard Good!
08:15 🔗 Cameron_D Yeah, I'm running 20 instances too
08:15 🔗 Hydriz but something slowed things down
08:15 🔗 Hydriz I am stopping already
08:15 🔗 Hydriz restarting soon
08:15 🔗 Cameron_D Yeah, things have slowed down
08:15 🔗 SketchCow Generate SSH Keys (http://help.github.com/msysgit-key-setup/)
08:15 🔗 SketchCow Install Git for Windows (http://help.github.com/win-git-installation/)
08:15 🔗 SketchCow Install XAMPP (http://www.apachefriends.org/en/xampp.html)
08:15 🔗 SketchCow Get the latest ThinkUp files from GitHub git clone git://github.com/ginatrapani/ThinkUp.git
08:15 🔗 SketchCow Make a MySQL database
08:15 🔗 SketchCow Edit httpd.conf to point to webapp directory
08:15 🔗 SketchCow Register the app with Twitter:
08:15 🔗 SketchCow Application Website: http://yourdomain.com/thinkup
08:15 🔗 SketchCow Application Type: Browser
08:15 🔗 SketchCow Callback URL: http://yourdomain.com/thinkup/plugins/twitter/auth.php
08:15 🔗 SketchCow Copy the Consumer key and Consumer secret
08:15 🔗 SketchCow Run the web-based install at http://yourdomain.com/thinkup/install/ and follow its steps
08:16 🔗 SketchCow Authorize ThinkUp to use your Twitter account
08:16 🔗 SketchCow Create a batch file to run crawler automatically
08:16 🔗 SketchCow A SNAP
08:16 🔗 closure whoh dude, everyone stopped at MySQL
08:16 🔗 yipdw it's sad when my first reaction to a wall of steps is "is there a Puppet script or Chef recipe for that"
08:17 🔗 yipdw means I've become lazy as hell
08:17 🔗 chronomex yipdw: friend of mine did a lot of work on Chef :)
08:18 🔗 yipdw chronomex: I like it a lot; been rolling it out at work
08:18 🔗 closure have you seen lockerproject.org? Similar concept, being done by a company and I know one of the guys.
08:19 🔗 likwid_ http://pastebin.com/DJrGrzT4 <- TweetDelete.rb , lets you archive and/or delete tweets from a twitter account
08:20 🔗 yipdw chronomex: it's probably the biggest thing that has made me not go completely insane over SELinux configuration, because I can shove all of that into recipes and comment just what the hell is going on
08:20 🔗 likwid_ current limitations: it's slow when you're destroying the tweets, processing at about 0.62 tweets/sec (but that does include inserting them into SQLite)
08:20 🔗 chronomex yipdw: heh, seriously
08:21 🔗 likwid_ it can use any DB that the Sequel gem can use, which includes SQLite, MySQL, Postgre, DB2, Oracle, etc. I will work tomorrow on a version that uses Ruby fibers to get better speeds. The limiting factor is the Twitter API (it is very slow)
08:21 🔗 SketchCow The thing is, yeah. Mysql.
08:21 🔗 SketchCow Why can't they do a flatfile?
08:22 🔗 SketchCow I mean, at most, let's say you're a fucking maniac
08:22 🔗 SketchCow You do 30,000 tweets
08:22 🔗 SketchCow Wow, what a maniac
08:22 🔗 SketchCow 30. thousand.
08:22 🔗 chronomex sqlite is a happy medium in most cases, flatfile is futureproof
08:22 🔗 likwid_ I have 15K
08:22 🔗 likwid_ tweets
08:22 🔗 chronomex 30k is a decent amount
08:22 🔗 SketchCow That's, like, I mean, probably a meg or two.
08:22 🔗 Cameron_D You cna only access the 3,200 most recent tweets can't you?
08:22 🔗 likwid_ another problem is that Twitter will only let you get the most recent 3,200
08:22 🔗 SketchCow Surely we need a massive database infrastructure.
08:22 🔗 likwid_ yes
08:23 🔗 likwid_ hmm, SQLite can dump to CSV easily
08:23 🔗 chronomex sqlite is plenty for most everything
08:23 🔗 SketchCow It needs some archiveteam
08:23 🔗 chronomex sms message store on android and iphone uses sqlite happily
08:23 🔗 db48x yea, sqlite is actually pretty awesome
08:23 🔗 likwid_ chronomex: except apps that are multi-user :)
08:24 🔗 chronomex likwid_: yeah, well, that's out of the defined use-case ;)
08:24 🔗 likwid_ chronomex: indeed, but I've seen lots of webapps using SQLite
08:24 🔗 db48x plus, the user doesn't have to install and set up a database server
08:24 🔗 SketchCow So, I propose a parallel version
08:24 🔗 SketchCow "DrinkUp"
08:25 🔗 SketchCow And we play around with it
08:25 🔗 yipdw does Twitter have some way of hooking into tweet feeds via XMPP/AMQP/whatever
08:25 🔗 likwid_ yeah, that will come tomorrow because it's 3AM here :)
08:25 🔗 yipdw ?
08:25 🔗 yipdw I guess I should RTFAPIDocs
08:25 🔗 likwid_ but I "solved" the 3,200 limit by making it remove all your tweets
08:25 🔗 closure yipdw: xmpp> hell no
08:25 🔗 likwid_ so then, theoretically, that "3,200" window slides back in time and you can collect all your data
08:26 🔗 chronomex likwid_: weird, but okay.
08:26 🔗 likwid_ chronomex: well, there is no other way to do it
08:26 🔗 chronomex suckass.
08:26 🔗 likwid_ chronomex: I am OK with this since I'm rebooting my public persona anyway
08:26 🔗 yipdw oh, twitter has a streaming thing
08:26 🔗 chronomex 00:22:22 <@SketchCow> You do 30,000 tweets
08:26 🔗 chronomex 00:22:24 <@SketchCow> Wow, what a maniac
08:26 🔗 likwid_ chronomex: but yeah, for most users it's not a workable solution
08:26 🔗 likwid_ haha
08:26 🔗 chronomex SketchCow: you have 12,000 tweets.
08:26 🔗 yipdw doesn't help for archival of past tweets, but going forward
08:26 🔗 SketchCow Yes
08:27 🔗 chronomex what a maniac!
08:27 🔗 SketchCow I am saying that even that much is nothing
08:27 🔗 likwid_ yeah Twitter actually has some crazy shit like the "Firehose"
08:27 🔗 SketchCow No mysql needed
08:27 🔗 likwid_ but you basically need to be DHS to get access to that
08:28 🔗 yipdw what, bloated and incompetent
08:28 🔗 chronomex creepy and overfunded.
08:28 🔗 likwid_ there is stuff like this: https://www.tweetscan.com/data.php but that is pulling from their own DB
08:28 🔗 SketchCow touching teenage girls
08:29 🔗 likwid_ what is the data mining company that starts with PH something?
08:29 🔗 likwid_ maybe it's just P.... they're in Palo Alto
08:29 🔗 yipdw Palantir
08:29 🔗 likwid_ yes
08:29 🔗 likwid_ I was considering, for about 10 seconds, working there
08:29 🔗 yipdw it seems to be popular amongst Stanford grads
08:29 🔗 yipdw but it seems that if you want a job in which you get to play with huge data sets
08:30 🔗 yipdw there must be better places to be
08:30 🔗 likwid_ eh
08:30 🔗 likwid_ they also have all the fun new toys
08:30 🔗 yipdw better meaning "I won't look back on this in 30 years and hate myself"
08:30 🔗 likwid_ yes, that
08:30 🔗 Hydriz Can I ask: If I have an rsync slot, and would like to upload another project, do I still have to ask?
08:30 🔗 db48x Hydriz: I don't think you're allowed to stop
08:30 🔗 likwid_ archive.org would be much more fun as a job for me anyway :)
08:30 🔗 Hydriz wait, what?
08:31 🔗 chronomex put em in different directories.
08:31 🔗 chronomex it'll be FINE.
08:31 🔗 Hydriz oh good haha
08:31 🔗 db48x Hydriz: ;)
08:31 🔗 yipdw don't stop archeeving, hold on to that feeling
08:31 🔗 likwid_ SketchCow: I read the article about scanning the braille Playboy. Cool stuff :)
08:32 🔗 Hydriz no, I mean if we are lets say finished with AnyHub
08:32 🔗 likwid_ (( http://ianews.wordpress.com/2011/08/17/scanning-a-braille-playboy/ ))
08:32 🔗 SketchCow Yes.
08:32 🔗 SketchCow (I guarantee the people in here.)
08:32 🔗 SketchCow (Have seen it)
08:34 🔗 likwid_ anyway, it got me thinking about converting stuff like that to Portable Embosser Format to allow electronic distribution of braille documents
08:35 🔗 likwid_ or, taking it a step further, making a PEF -> plaintext converter so that you could eventually text->speech that stuff
08:44 🔗 yipdw haha, the downside of cache-busting query-string parameters
08:45 🔗 yipdw splinder has a TON of them for images
08:47 🔗 db48x heh
08:47 🔗 yipdw so as a result, journal.splinder.com is at 48,000 PNGs and counting
08:47 🔗 yipdw or something
08:48 🔗 chronomex jeepers
08:48 🔗 yipdw it's been downloading for almost three das
08:48 🔗 yipdw days
08:49 🔗 yipdw actually, wait a sec, it's not cache-busting
08:50 🔗 chronomex different releases?
08:50 🔗 yipdw it's 47,218 PNGs all with unique URLs modulo query string
08:50 🔗 yipdw it looks like...a bunch of shrunken avatars
08:50 🔗 yipdw oh, christ
08:50 🔗 yipdw I think this might end up downloading a sizable chunk of all Splinder users' avatars
08:50 🔗 db48x heh
08:50 🔗 yipdw because journal.splinder.com is like the Splinder dev journal or something
09:53 🔗 Hydriz Erm, any professional helper here?
09:54 🔗 chronomex what kind of professional?
09:55 🔗 Hydriz I mean, installing wget-warc
09:55 🔗 Hydriz I am grabbing from a shell provider, but want to do it on my personal comp
09:55 🔗 chronomex ah.
09:55 🔗 Hydriz but it always seems to fail
09:56 🔗 Hydriz can you help?
09:57 🔗 Hydriz I receive this error: configure: error: --with-ssl was given, but GNUTLS is not available.
09:57 🔗 Hydriz and try installing GNUTLS
09:57 🔗 chronomex apt-get install gnutls-dev ?
09:58 🔗 Hydriz hmm
09:58 🔗 Hydriz lemme try
10:02 🔗 Hydriz looks okay for now...
10:02 🔗 Hydriz yeah! Thanks
10:02 🔗 Hydriz :)
10:02 🔗 chronomex :D
10:55 🔗 ersi !likes
10:55 🔗 ersi oops :)
11:30 🔗 DoubleJ ypidw: Same with one of the ones I'm downloading. I have hundreds of thousands of files from files.splinder.com.
11:44 🔗 Nemo_bis I lost the connection for some 20 min, now it's downloading very little, how do I know if a process is just long or stale?
15:12 🔗 ersi http://adamsworld.name/copy_calc.php lol at that message
15:13 🔗 ersi "Website Offline, No Cached Version Available"
15:13 🔗 ersi and they're a.... caching service.
15:13 🔗 ersi good game, asshats
15:44 🔗 rude___ HULLO
18:10 🔗 Nemo_bis why are processes so different in a "pidstat -u -T CHILD -C dld-client" ? http://p.defau.lt/?y9WaTFshFkhXK868M0zVoQ
18:10 🔗 Nemo_bis Does it depend on whether their children are still alive?
18:15 🔗 closure woot! I've passed the 100 thousand users downloaded mark on splinder
18:15 🔗 closure ndurner is gonna pass me tho
18:15 🔗 Nemo_bis yep, he's accelerating :-p
18:16 🔗 Nemo_bis closure, can you answer my first question above?
18:16 🔗 closure no idea
18:17 🔗 Nemo_bis closure, I mean Nemo_bis> I lost the connection for some 20 min, now it's downloading very little, how do I know if a process is just long or stale?
18:17 🔗 closure oh, sure, I just strace the wget to see what it's doing
18:17 🔗 Nemo_bis closure, eh, if only I knew the pid to do it selectively
18:18 🔗 Nemo_bis or is there a way to do it quickly on all of them and automatically find the stale ones?
18:18 🔗 closure ps -fax
18:36 🔗 Nemo_bis closure, thanks, looks like they're working
18:37 🔗 closure yeah, I've seen no full stalls, just some big blogs downloading for days on end
18:37 🔗 Nemo_bis wget is remarkably stable even on crazy connection, better than expected :-D
18:38 🔗 Nemo_bis closure, is there any way to tell wget to use more memory and write to disk less often?
19:28 🔗 closure Nemo_bis: maybe you want to ionice -c 3 wget?
19:30 🔗 Nemo_bis closure, interesting, does it work on all running processes?
19:32 🔗 closure sure
19:39 🔗 Nemo_bis closure, sure? « -p pid Pass in process PID(s) to view or change already running processes. If this argument is not given, ionice will run the listed program with the given parameters.»
20:45 🔗 alard Hi Splinder downloaders: Nothing urgent, but if you want full credit for your downloads, please do a git pull. (No restarts necessary.) It turns out that recent versions didn't report the size of the media.warc.
20:46 🔗 db48x2 heh, oops
20:50 🔗 Nemo_bis closure, looks better now, I think http://p.defau.lt/?9ahUP_jXIxJTSIWKX_KDZw (300 s intervals)
20:51 🔗 Nemo_bis alard, yes, it's something like 1/4 of the real total
20:51 🔗 alard The media bit is well, somewhat smaller than it should be.
20:51 🔗 Marcelo What's happening?
20:51 🔗 alard Hi Splinder downloaders: Nothing urgent, but if you want full credit for your downloads, please do a git pull. (No restarts necessary.) It turns out that recent versions didn't report the size of the media.warc.
20:51 🔗 alard Marcelo:
20:54 🔗 Marcelo I'll go back to midpSSH and continue downloading. Bye.
21:58 🔗 closure hah, told you it was underreporting :P
22:08 🔗 alard closure: Heh, yes. (But you were not convincing enough. :)
22:40 🔗 yipdw huh, this is a fun anyhub-grab error
22:41 🔗 yipdw 2011-11-16 16:35:37 URL:http://f.anyhub.net/4o0J [84178/84178] -> "/dev/null" [1]
22:41 🔗 yipdw Cannot write to `/dev/null' (Success).
22:50 🔗 alard yipdw: I had that too, today, multiple times.
22:59 🔗 underscor ASCII a stupid question, get a stupid ANSI!
22:59 🔗 closure EBCDICK
23:08 🔗 SketchCow The fun's begun - uploading gigabytes of cool game convention videos to the GDC Vault
23:08 🔗 SketchCow Everything I upload goes into their free section.
23:09 🔗 underscor Awesome!
23:10 🔗 underscor Only 5k anyhub users left!
23:23 🔗 Nemo_bis SketchCow, how do I upload Splinder data to the batcave?
23:24 🔗 SketchCow You ask me for a rsync slot.
23:24 🔗 SketchCow Dod you have one?
23:24 🔗 SketchCow Do you?
23:25 🔗 Nemo_bis SketchCow, no
23:25 🔗 Nemo_bis I need one for donbex also if possible
23:26 🔗 Nemo_bis (a friend who's downloading Splinder)
irclogger-viewer