#archiveteam 2011-09-13,Tue

↑back Search

Time Nickname Message
00:29 🔗 godane episode 4x07 is up of hak5
01:04 🔗 dashcloud how's this look? http://pastebin.com/BwEgbDT1
01:13 🔗 chronomex how about some line breaks
01:16 🔗 dashcloud I was told one giant line of text is what was wanted, so that's how I re-formatted it
01:17 🔗 dashcloud if you've got an example or a sample I can look at, that would be appreciated
01:17 🔗 SketchCow Almost got it.
01:17 🔗 SketchCow I don't literally mean HEADER:
01:18 🔗 SketchCow I want it like this, no line reaks
01:18 🔗 SketchCow But HEADER isn't needed, page numbers not needed.
01:27 🔗 dashcloud how's this: http://pastebin.com/w85muBcN ?
01:33 🔗 SketchCow Big K Magazine, April 1984. Contents. Games Programs: ROCKET for VIC 20, BOMB RUN for ORIC, DEMON DRIVER for COMMODORE 64, DOWN FALL for BBC Model 8, ESCAPE for SPECTRUM. SOFTWARE REVIEWS: Charlie Nicholas reviews for us. HARDWARE: Wonderful Widgets, Brilliant Bodges- A Cheapo Epro, Goad Your Code the 6502 Way, Squaring Up- Atari v. Acorn 91. FEATURES: Do you Sincerely Want to be Rich? ...
01:33 🔗 SketchCow See what I did there?
01:33 🔗 dashcloud okay- got it
01:34 🔗 SketchCow I guess Games Programs should be GAMES PROGRAMS:
01:38 🔗 dashcloud What about the free-standing bits at the end?
01:42 🔗 SketchCow Don't go crazy
01:42 🔗 dashcloud okay then- http://pastebin.com/FDBHL2MX
01:43 🔗 SketchCow Yes.
01:43 🔗 SketchCow proceed
01:44 🔗 dashcloud move on to the next one? or did you want more stuff captured from the contents page?
01:45 🔗 SketchCow I mean go on, do it all
01:45 🔗 SketchCow The style is good
01:45 🔗 SketchCow Do this issue, and move on to next issue, this will be good.
01:45 🔗 dashcloud okay
02:16 🔗 dashcloud did you also want the boilerplate at the bottom of the page?
02:23 🔗 SketchCow No
02:26 🔗 dashcloud okay- think I got the first one done then: http://pastebin.com/79nPpTLk
02:32 🔗 dashcloud I showed your post about the pirate radio archive to a guy I know online who's into it, and he pointed me to http://radio-airchecks.nl/ , which he says has 500 GB at least of pirate radio recordings
02:42 🔗 godane just watch a clip of glenn beck on black tom explosion
02:42 🔗 godane http://en.wikipedia.org/wiki/Black_Tom_explosion
04:56 🔗 SketchCow Well, even though it's taking a billion years, I am uploading those 80 Microcomputers.
05:20 🔗 SketchCow Just blew in 61 issues of Commodore Format magazine, which was dedicated to the Commodore 64.
05:20 🔗 SketchCow http://www.archive.org/details/commodore-format-magazine
05:20 🔗 SketchCow Should be ready for perusal in an hour or so.
05:50 🔗 SketchCow just thought I say hi and let you know I haven't forgotten you. One day... in some way... you will be repaid. Dishonesty is too nice a word for you... Have a happy holiday season... NOT.
05:50 🔗 SketchCow My e-mail is awesome
05:50 🔗 SketchCow This is a guy whose hard drive I still have
05:50 🔗 SketchCow Some of you have noticed my slow turnaround
05:50 🔗 SketchCow He turned abusive
05:51 🔗 SketchCow Not surprisingly, I think you'll understand, his turnaround became slower
05:51 🔗 SketchCow So we're now in xeno's paradox
05:54 🔗 godane ShetchCow: i readed about your distriwiki
05:54 🔗 godane couldn't that be done using a git/mecurinal like vcs
05:57 🔗 SketchCow It could be done a ton of ways.
05:57 🔗 SketchCow It'd be a module.
05:57 🔗 godane i think my linux source dvd will be of some use
05:58 🔗 godane i have full distro that will be able to rebuild it self
05:58 🔗 godane websites, sources tarballs ( recompressed to .tar.lzma to save space), and repos of projects
06:01 🔗 godane i also use dokuwiki cause slitaz doc website does
06:02 🔗 godane dokuwiki puts all docs in plan text :-D
06:02 🔗 godane the history is a problem when the users of changed history doesn't exist though
06:04 🔗 SketchCow Not true
06:04 🔗 SketchCow It means there has to be a shared update process
06:04 🔗 SketchCow And it means you will have race conditions
06:05 🔗 SketchCow And those are all problems that need fixing.
06:05 🔗 godane ok
06:06 🔗 godane but all changes should be able to be reverse anytime?
06:06 🔗 godane i only think git or mecurinal cause you can just branch off the master/default branch
06:26 🔗 ersi not often you see SketchCow talking with himself
06:29 🔗 ersi So.. this is the fifth day of my instructables.com mirroring
07:07 🔗 SketchCow It' a big one.
07:07 🔗 ersi indeed
07:19 🔗 ersi It's growing slowly.. I'm up at 28GB now
07:20 🔗 ersi that's 5-6GB/day
07:21 🔗 ersi someone here had downloaded 40GB "once upon a time).. I'm atleast 1-3 days away from that
07:48 🔗 SketchCow http://vimeo.com/28976327
07:50 🔗 ersi Ooh
07:52 🔗 ersi Hm, I'd do either 6502 or Tape - but Arcade would be interesting as well, even though that's atleast a little covered
07:53 🔗 db48x2 SketchCow: interesting video. I thought the tense music was odd though
07:53 🔗 db48x2 I laughed at the tagline for the Tape documentary though, which is good :)
07:53 🔗 ersi It added atmosphere
07:54 🔗 SketchCow This whole thing is completely off-kilter.
07:55 🔗 db48x2 yea, the atmosphere felt wrong somehow
07:56 🔗 ersi i liked it
07:56 🔗 ersi felt, human.
07:58 🔗 SketchCow It is intentionally wrong.
07:58 🔗 SketchCow You won't forget it soon, will you.
08:01 🔗 ersi Ha, strategic
08:02 🔗 SketchCow The whole thing is strategic.
08:02 🔗 SketchCow It appeals to a certain kind of person.
08:02 🔗 SketchCow A person who would give me hundreds of dollars and not see a thing for years.
08:03 🔗 SketchCow That's not reddit people.
08:03 🔗 SketchCow :)
08:03 🔗 SketchCow It also gets weirder the more times you play it.
08:04 🔗 db48x2 heh
08:06 🔗 SketchCow Doesn't it.
08:07 🔗 ersi It does.
08:09 🔗 SketchCow You know those guys who make something filmy and then run around showing you their stuff and watching you and quizzing you on what you think?
08:09 🔗 SketchCow I ain't one of those guys.
08:10 🔗 SketchCow But I will say, I accounted for the liking two out of three.
08:10 🔗 SketchCow You can invest with premiums in two
08:11 🔗 SketchCow or you can invest in all three for slightly less than normal all three.
08:11 🔗 db48x2 ah, I was mislead by the "beta" designation
08:12 🔗 SketchCow Well, I like reactions.
08:12 🔗 SketchCow But I don't seek it out.
08:13 🔗 SketchCow Either people will invest, and I'll hit my goal, or they won't.
08:13 🔗 SketchCow And then I merely have to archive forever
08:13 🔗 db48x2 heh
08:14 🔗 SketchCow Beta is merely my worry of it not rendering.,
08:14 🔗 db48x2 ah
08:15 🔗 db48x2 so which of the three would you prefer to do?
08:16 🔗 SketchCow All
08:16 🔗 ersi otherwise he would have promoted only one :) (I think)
08:17 🔗 SketchCow ha ha
08:17 🔗 SketchCow Uploaded it to kickstarter page (preview)
08:17 🔗 SketchCow I just love it
08:17 🔗 SketchCow What a weird video
08:23 🔗 * db48x2 yawns
09:07 🔗 db48x2 hrm
09:07 🔗 db48x2 I can't find my book of stamps
10:09 🔗 db48x2 I had it here somewhere not even a year ago
10:41 🔗 kin37ik finally, my cap shall refresh at midnight!
10:42 🔗 db48x2 heh
10:44 🔗 kin37ik gah! i hate this stupid shake thing in windows7
10:45 🔗 db48x2 shake thing?
10:45 🔗 kin37ik yeah, where you grab the window panel
10:45 🔗 kin37ik you shake it left and right like twice and it minimizes all the windows except the one your dragging
10:47 🔗 db48x2 oh, right
10:47 🔗 db48x2 why do you hate it?
10:47 🔗 kin37ik because i have 3 monitors, so, when i drag something across, it assumes im doing the shake thing
10:47 🔗 kin37ik and minimizes everything when i like ot have it up where it is
10:48 🔗 db48x2 huh, I don't have that problem with my three monitors
10:48 🔗 kin37ik you using eyefinity though?
10:48 🔗 db48x2 hmm, not at the moment
10:48 🔗 kin37ik ahh, se eim using eyefinity
10:48 🔗 kin37ik see im*
10:50 🔗 db48x2 lol
10:50 🔗 db48x2 I turned on eyefinity and it's got my monitors arranged vertically
10:52 🔗 kin37ik you should be able to drga the monitors around on the plotter thing to put them right lol
10:52 🔗 db48x2 no, I had to disable eyefinity and set it up again
10:52 🔗 kin37ik O.o
10:52 🔗 db48x2 then it let me choose between 1x3 and 3x1
10:53 🔗 kin37ik ahh yeah
10:53 🔗 kin37ik ive done 5 monitor setups with eyefinity lol
10:54 🔗 db48x2 ok, now I've got it set up "right"
10:54 🔗 kin37ik awsome!
10:54 🔗 db48x2 dragging windows around doesn't trigger the shake gesture though
10:54 🔗 db48x2 even across monitor boundaries
10:54 🔗 kin37ik for mine it does oddly enough
10:54 🔗 kin37ik im not sure how i can turn it off
10:54 🔗 db48x2 weird
10:55 🔗 kin37ik though im using an XFX card and sometimes i wonder how shoddy the drivers are
10:56 🔗 kin37ik already got a problem with my display port because of the cards bios and they refused to give me an updated bios
10:57 🔗 db48x2 fun
10:57 🔗 db48x2 well, I have to go back to my old settings
10:57 🔗 db48x2 eyefinity is ok for games, but terrible for a normal windows desktop
10:57 🔗 db48x2 and also my monitors are not all the same resolution
10:59 🔗 db48x2 there, back to normal
11:00 🔗 db48x2 well, except that all my windows are on the wrong displays :)
11:00 🔗 Cameron_D Arg, OCD overload
11:00 🔗 Cameron_D Windows are not allowed to move
11:00 🔗 db48x2 :)
11:01 🔗 db48x2 they should all be maximized
11:01 🔗 db48x2 or at least almost all of them should be maximized
11:02 🔗 Cameron_D main screen has my main program maximised and my 2nd screen has IRC and whatever other chat windows laid out in a way where I can see most of them
11:27 🔗 dnova SketchCow: nice shortened url in your pitch vid :P
11:28 🔗 dnova the page doesn't appear to be up yet. are you proposing all three of those or letting people choose one?
11:29 🔗 dnova because, fuck, I want all three
11:42 🔗 tef random question: how do you store your archives? warc? arc? tgz of directory?
11:43 🔗 ersi I store them in a gigantic .derp
11:43 🔗 ersi all files appended after each other
11:44 🔗 tef do you have a .herp file with the offsets
11:48 🔗 kin37ik lol
11:48 🔗 ersi No, that's not derpy or herpy at all
11:49 🔗 tef :(
11:49 🔗 tef just I notice from http://www.archiveteam.org/index.php?title=Wget_with_WARC_output you're using the old version of warctools
11:50 🔗 tef (which is well, unpleasant)
11:50 🔗 ersi I just wget, without WARC
11:50 🔗 tef ah ok
11:50 🔗 tef it is just I am the person who is writing the new one
11:50 🔗 ersi also, you're free to uphax the code for warc support
11:51 🔗 ersi alard wrote that warc support a little while ago
11:51 🔗 tef well the problem is that we use python now instead of C, so it isn't as easily hacked into wget
11:52 🔗 tef but this is why I was asking about warc files
11:52 🔗 ersi we who what?
11:52 🔗 tef oh
11:52 🔗 ersi and are you saying WARC changes frequently?
11:52 🔗 tef I work for the company that wrote warc-tools (the c lib on google code)
11:53 🔗 Cameron_D I'll take a python WARC library
11:53 🔗 tef We no longer use or maintain it, and we're currently using a python library instead
11:53 🔗 ersi ah-ha
11:53 🔗 tef sorry, yeah I should have owned up to that earlier
11:55 🔗 tef Cameron_D: I could probably knock up a wget like script that uses it
11:55 🔗 tef Cameron_D: we use it in production but heh, my attention has been on the bits that use the library rather than the library itself
11:55 🔗 alard Hi all.
11:55 🔗 tef but I have time alloted to deal with support issues for it
11:55 🔗 tef http://code.hanzoarchives.com/warc-tools/overview
11:55 🔗 ersi o/ alard
11:55 🔗 alard tef: yes, wget-warc uses the old c version (which seems to work pretty well).
11:59 🔗 ersi meh, as long as it doesn't produce unusable WARC archives
11:59 🔗 tef not so far
11:59 🔗 tef (heh)
11:59 🔗 tef but it turns out lots of warcs are a bit special
12:00 🔗 ersi oh?
12:00 🔗 tef I found one with unix line separators and gzipped fully rather than crlf and each record gzipped
12:00 🔗 tef and there are a bunch of pre 1.0 ones floating around
12:00 🔗 db48x2 that's always fun
12:01 🔗 Cameron_D tef, cant look through the code at the moment, but is there a usage example of sorts?
12:01 🔗 tef Cameron_D: there are some scripts in the repo for opening/reading warcs and arc2warc conversion
12:02 🔗 tef apologies for the lack of documentation. we're a small company and we're a little rushed off our feet at the moment
12:03 🔗 db48x2 it's ok
12:03 🔗 tef i'll see if I can merge in a python wget example to it
12:03 🔗 tef I have some code knocking around for that that doesn't use wget https://github.com/tef/codesamples/tree/master/pyget
12:04 🔗 ersi Mmmmh, 8.4GB memory usage from wget
12:04 🔗 tef doesn't use warctools either
12:04 🔗 Cameron_D tef, thanks, I'll take a look
12:04 🔗 tef Cameron_D: if you have any questions about warctools email me directly at thomas.figg@hanzoarchives.com
12:05 🔗 kin37ik hmm, i got a question about Wget actually
12:05 🔗 tef I have *some* time alloted to deal with support/features
12:05 🔗 ersi about wget, or wget and warc?
12:05 🔗 kin37ik just wget on it's own
12:05 🔗 kin37ik what i want to know is
12:05 🔗 kin37ik say if im poking a url, for example www.example.com/millenium/0001/ and it has a number heirarchy for directories right
12:06 🔗 ersi Fortunecity? :)
12:06 🔗 kin37ik is there a script i can use to incrementally increase the number to a specified limit and stop when it hits the number or?
12:06 🔗 kin37ik yeah
12:06 🔗 db48x2 bash
12:06 🔗 ersi bash would be your friend, yes
12:07 🔗 db48x2 www.example.com/millenium/{0001..9999}/ should do the trick. it'll be longer than the maximum command line length
12:07 🔗 Schbirid SketchCow: that last shot in the kickstarter vid is kinda awkward
12:07 🔗 tef for i in `seq ... ...`; do ....; done
12:07 🔗 kin37ik db48x2: cheers mate (:
12:09 🔗 db48x2 yw
12:09 🔗 tef ersi: btw, which features of wget are the most useful to you?
12:10 🔗 godane a compress option i think is need for wget-warc
12:10 🔗 tef (my boss is happy for me to make a simplified wget example for the new warctools)
12:10 🔗 godane only cause right now it only saves to as gzip/tar.gz
12:11 🔗 db48x2 godane: there is one
12:11 🔗 tef per record compression ?
12:11 🔗 ersi tef: the regular mirror switch, convert links to local ones and keep original (-kK) http/ftp support
12:11 🔗 tef cool
12:11 🔗 db48x2 godane: --no-warc-compression
12:11 🔗 Schbirid --content-disposition is crucial for me
12:12 🔗 db48x2 --random-wait -EkKp --protocol-directories -np --follow-ftp
12:12 🔗 godane was talking about about changing the it to bz2 or .lzma
12:13 🔗 tef gzip is pretty de-facto for warcs
12:13 🔗 godane ok
12:13 🔗 db48x2 oh, and --user-agent, but that's easy to do
12:14 🔗 godane thought it would be nice to add lzma so you can save more space
12:14 🔗 tef this is really helpful
12:15 🔗 tef ersi: most stuff does re-writing after creating warc files
12:15 🔗 tef the idea being the warc record being an exact snapshot of the wire traffic
12:15 🔗 tef (near enough)
12:15 🔗 db48x2 tef: honestly I think it'd be easier to integrate the python library into wget
12:15 🔗 db48x2 tef: that reminds me
12:15 🔗 tef hmm
12:16 🔗 db48x2 tef: alard was working on a way to feed a warc into wget and have wget output the set of mirrored directories
12:16 🔗 tef ah I see
12:16 🔗 tef warc unpacker
12:17 🔗 db48x2 yea
12:17 🔗 db48x2 it could do the -k (and -K) stuff
12:17 🔗 tef well
12:17 🔗 tef in a warc record
12:17 🔗 tef you can have request/response
12:18 🔗 tef as well as conversion records
12:18 🔗 db48x2 yea
12:18 🔗 tef so the -K stuff would be writing those in some fashion
12:18 🔗 db48x2 I had wanted to put conversion records into the warc
12:18 🔗 db48x2 it's a bit tricky, so I haven't done it yet
12:18 🔗 tef (we strip transfer-chunked and content-encoding)
12:21 🔗 tef it seems most of the wget options you guys use are about unpacking/rewriting the content i.e -EkK
12:21 🔗 tef and a few for navigation i.e -p -np --follow-ftp
12:23 🔗 tef so there is less need to clone wget if wget can read from warcs via some method (i.e a proxy)
12:23 🔗 ersi and generally wget's link traversing (mirroring)
12:26 🔗 tef i'm not sure what that meansin specific - do you mean keeping an existing archive up to date?
12:27 🔗 tef or do you mean the scope of links that are checked
12:27 🔗 ersi the scope of links fetched
12:27 🔗 tef ah
12:27 🔗 db48x2 yea, wget does a lot of parsing of html and css
12:27 🔗 ersi it seems to do the job well of going deeper
12:28 🔗 ersi and when it's getting page recreciuits it's awesome
12:28 🔗 ersi oops, what the hell happened there
12:28 🔗 ersi page requisits(sp?)
12:28 🔗 tef to some extent I think it would be better trying to play to the strengths of being written in python - hackable, rather than apeing wget entirely
12:28 🔗 tef i.e scriptable for those more awkward things rather than having to resort to bash :-)
12:29 🔗 ersi Nothing wrong with blunt tools, even though I like python
12:29 🔗 ersi ;D
12:29 🔗 tef yeah but you have a very good blunt tool
12:30 🔗 tef ersi: thanks again for taking the time to explain this stuff
12:30 🔗 tef fwiw both I and my boss have a soft spot for the work archive team does so we'd like to help out where we can, esp re warctools
12:31 🔗 tef </corporate_shill>
12:34 🔗 tef anyway, i'll shut up now and i'll talk again when i've got something to show for it
12:34 🔗 tef cheers
12:41 🔗 db48x2 heh
12:41 🔗 db48x2 you can talk whenever you like
12:45 🔗 ersi tef: no prob of course :)
12:45 🔗 tef db48x2: I hate that thing where someone comes in with a driveby idea
12:46 🔗 db48x2 :)
12:46 🔗 ersi well, it's different from getting feedback when one is more likely to do something about it
12:46 🔗 ersi and most of us here have a softspot for IA, so WARC is closeby in our hearts
12:46 🔗 ersi even if they... care about robots.txts
12:47 🔗 ersi (I see why though, sucks getting blocked or taken unseriously)
12:47 🔗 ersi Booya! One more bug/defect reported~ then I'll look extra productive
13:42 🔗 Schbirid http://nationalmap.gov/historical/
14:30 🔗 db48x Schbirid: sweet
14:36 🔗 ersi Bluh, ..instructables.. /keyword-iphone/keyword-easy/index.html
14:40 🔗 db48x Schbirid: are you able to download any maps from that?
14:41 🔗 db48x the links in the 'Download GeoPDF' column all just point back to the search results
14:42 🔗 db48x the map search works though
14:43 🔗 SketchCow Morning.
14:44 🔗 db48x hello SketchCow
14:45 🔗 Schbirid sorry didnt try
14:45 🔗 db48x downloading maps is very slow
14:45 🔗 db48x 25kBps :)
14:46 🔗 Schbirid it was announced today so surely a lot of traffic
14:46 🔗 db48x yea
16:05 🔗 SketchCow Hey, so I wrote the 80 micro archive that has some "offline" and asked for them
16:05 🔗 SketchCow Less than 24 hours later, here they come.
16:14 🔗 DFJustin http://www.abandonware-magazines.org/index.php
16:16 🔗 db48x does anyone have a link to a pdf on an https server?
16:16 🔗 db48x I have to test specifically that combination
16:16 🔗 SketchCow That's quite a link, DFJustin
16:31 🔗 closure underscor: today would be a good day to fix your olduse.net shell box. (on Boing Boing)
16:36 🔗 closure underscor: oh, it works again, NM
18:08 🔗 SketchCow http://kck.st/jasonscott
18:15 🔗 Soojin cool, will brute force spread :)
18:17 🔗 dnova big jump from $10 to $100
18:19 🔗 SketchCow Yes.
18:24 🔗 dnova Will these each be shorter/less comprehensive than BBS/Getlamp?
18:28 🔗 SketchCow No.
18:29 🔗 dnova wow
18:42 🔗 chronomex fucking scanner is misbehaving
18:42 🔗 chronomex today is an angry technology day
18:42 🔗 chronomex actually it's an angry chronomex day
18:43 🔗 chronomex SketchCow: radio silence, btw
18:45 🔗 sep332 grr technology
18:47 🔗 SketchCow ALexis got very sick
18:47 🔗 SketchCow She's in bed since Friday
18:48 🔗 chronomex oh dear
18:48 🔗 chronomex send her my regards
18:49 🔗 SketchCow So you're not being ignored.
18:52 🔗 chronomex ok
18:54 🔗 db48x I hope "RISE OF THE METADATA WARRIOR" will be recorded; the title is awesome
18:56 🔗 josephwdy Who is ALexis ?
18:57 🔗 chronomex SketchCow's boss at IA
18:58 🔗 dnova - Create a public, museum-like archive of 3D Porch in about 30 days. This museum will probably just have 500 or so photos.
18:58 🔗 dnova damn
18:58 🔗 rabidabid hello everybody
18:58 🔗 rabidabid i dunno if it's significant enough for you guys but I'm just throwing it out there
18:58 🔗 rabidabid it's a site where you can upload 3D photos
18:58 🔗 rabidabid it's not very big/popular and I don't think it's been around very long
18:58 🔗 rabidabid so uh, there's this site called 3D Porch which might be shutting down http://3dporch.com/
18:58 🔗 dnova 50,000 items, 7 files per
18:58 🔗 dnova :|
18:58 🔗 dnova I wonder how much it costs
18:59 🔗 SketchCow Someone please grab it
19:00 🔗 dnova gonna try to get in touch with him
19:00 🔗 chronomex grab first if it's that small
19:04 🔗 dnova man some people do NOT know how to use 3D cameras
19:26 🔗 dnova wish I could do higher than "project backer"
19:27 🔗 dnova I'm surprised tape is more popular than arcade at the moment
19:28 🔗 Schbirid i am glad :)
19:28 🔗 dnova I'm on that one
19:30 🔗 SketchCow It's a good and interesting way to get opinion. :)
19:31 🔗 SketchCow Ok, off to broadway
19:31 🔗 SketchCow seeing a musical for my birthday
19:31 🔗 dnova happy birthday! have fun
19:40 🔗 ersi I'd like Tape over Arcade as well
19:44 🔗 dnova well, buy it now!
19:44 🔗 dnova receive it in 4 years or so :)
19:46 🔗 alard There are 46425 photos on 3dporch, I think (the photos on the 'popular' lists). I have a list of the photo ids, downloading them now, unless someone else is doing that too.
19:47 🔗 dnova each one is 7 files
19:47 🔗 dnova + metadata if you care
19:47 🔗 alard 7? I have 6.
19:48 🔗 alard .jps
19:48 🔗 alard .left.jpg
19:48 🔗 alard .mpo
19:48 🔗 alard .redcyan.jpg
19:48 🔗 alard .right.jpg
19:48 🔗 alard .wiggle.gif
19:48 🔗 dnova missing .sbs
19:48 🔗 dnova oh nm
19:48 🔗 dnova those just rearrange left and right
19:48 🔗 dnova you're right
19:49 🔗 alard I am missing the wiggle.thumb
19:49 🔗 dnova what kind of speed are you getting
19:49 🔗 alard 4MB/s
19:49 🔗 dnova nice. should be quick work
19:49 🔗 alard It's amazon, so as fast as I can.
19:51 🔗 dnova how did you get all of the IDs?
19:52 🔗 alard I grabbed the 'popular' pages and extracted the IDs.
19:53 🔗 alard So I only have the popular page stuff, but maybe that's all there is?
19:53 🔗 alard Everything is popular?
19:56 🔗 dnova you can go to each type of camera
19:56 🔗 dnova and cross check
19:56 🔗 Coderjoe hmm
19:56 🔗 Coderjoe looks like IDs are just [0-9a-z]{4}
19:56 🔗 dnova caps also but yes
19:57 🔗 Coderjoe hahah
19:57 🔗 Coderjoe http://3dporch.com/b6gp
19:58 🔗 Coderjoe i have not yet encountered an uppercase letter in the ids
19:58 🔗 dnova alard: I really wish I knew how to do all that :|
19:59 🔗 dnova Coderjoe: go to nintendo 3ds there are a bunch on the first page
19:59 🔗 Coderjoe ah
19:59 🔗 rabidabid i think my favorite is this one http://3dporch.com/4gro
20:00 🔗 Coderjoe [0-9A-Za-z]{4} is enough for 14.8M IDs
20:01 🔗 Coderjoe not seeing much in the way of metadata
20:01 🔗 dnova hey, author replied
20:01 🔗 dnova er, owner
20:01 🔗 dnova I am really honored that you would pick my site to archive.
20:01 🔗 dnova Right now all the images are hosted on S3, which doesn't provide a convenient gunzip tool. The total data is about 50GB.
20:01 🔗 dnova Also, if you crawled 3D Porch, you'd miss a lot of anonymous uploads (about 50-80% of the site's content).
20:01 🔗 dnova I think the ideal would be for me to generate a static HTML version of the site, zip that and send it to you, and then let you fetch the individual S3 assets from that HTML. Then when you're done, I'll delete my S3 store.
20:01 🔗 Coderjoe viewcount and creator
20:01 🔗 dnova What do you think?
20:01 🔗 alard Cool!
20:02 🔗 dnova should I have him do that?
20:03 🔗 alard Yes, I think that would be very helpful. Getting a list of the ids is key, having the metadata in the html files is even better.
20:04 🔗 dnova replied
20:04 🔗 dnova no wonder he can't afford to keep it up if it's all on s3
20:13 🔗 sundown do ip registries such as ripe, arin etc offer their whois databases to the public?
20:14 🔗 sundown the content returned in response to whois queries
20:16 🔗 Schbirid i am currently trying to get http://who.is to stop showing an archived version of a site of mine where i used my full name so i hope not .p
20:16 🔗 sundown ip assignment whois or domain whois?
20:16 🔗 Schbirid many lookups will actually show you legalese about how youare not allowed to store that information iirc
20:16 🔗 Schbirid sorry
20:16 🔗 Schbirid domain
20:17 🔗 Schbirid <-stooopid
20:17 🔗 sundown do you know of whois.sc?
20:17 🔗 Schbirid domaintools.com nowadays
20:17 🔗 sundown yes
20:17 🔗 sundown they compile that information and sell it for commercial gain
20:17 🔗 Schbirid yeah
20:21 🔗 Schbirid nighty!
20:23 🔗 sundown is anyone interested in starting a project to archive ip assignment data, domain whois, and dns history?
20:26 🔗 sundown would that be in accord with the archive team philosophy? the intention is to make available this data to the world for free
20:27 🔗 ersi the philosophy is do whatever, aslong as you're doing something
20:28 🔗 ersi I'd be a bit worried about doing that though, there's a fuckload of fucktards who will spam you to oblivion if you keep that kind of data and say you do online AND provide it
20:30 🔗 sundown who would have reason to be angry? (other than domaintools, heh ;)
20:33 🔗 sundown i think i'll work independently and write about my progress on archive team wiki
20:36 🔗 DFJustin verisign might not be too happy
20:49 🔗 chronomex verisign can suck it
irclogger-viewer