[01:52] uploading video #200 [01:52] yay [01:58] excellent [02:07] holy crap- this must be the finest job posting ever: http://blogs.valvesoftware.com/abrash/ [02:08] just got nfs access to the imagedump server for wikimedia foundation [02:08] those will be going up soon on a.o [02:08] congrats! [02:09] thanks :) [02:09] Holy carp, is THAT where Abrash wound up? [02:10] after 14 years he came back there it looks like [02:12] 204.9.55.82:/z/public/pub/wikimedia/dumps 157T 31T 126T 20% /mnt/dumps [02:12] 204.9.55.82:/z/public/pub/wikimedia/images 143T 16T 126T 12% /mnt/images [02:12] wheeee [02:18] underscor: excellent! [02:19] i've got hundreds of photos there [02:19] :) [02:19] i was hoping they'd outlive wikimedia, and it seems it will [02:19] next step it to write ingestion logic to get it all into archive.org [02:19] s/it/is/ [02:19] (hundrds on the commons, that is) [02:19] underscor: good luck, and well done on scoring those [02:20] thanks [02:20] Wow, that's...a big array. [02:21] Two of them. [02:21] yeah [02:21] the box has something like 480TB on it [02:22] Where box == rack, I'd imagine. [02:23] it's all on one "machine" [02:23] connected over fibrechannel disk enclosures [02:24] So something like a Ceph cluster? Okay, makes sense. [02:59] Well, my archives con today reinforced how awesome AT is [02:59] So, go you guys o/ [03:01] shaqfu: marac? [03:01] mistym: ...how'd you know? [03:02] shaqfu: Seen a bunch of people I follow twittering it up today. Wasn't there myself. [03:02] mistym: Yep, was there today/tomorrow [03:02] mistym: Didn't know we had another traditional archivist in the room [03:04] shaqfu: Yep! For a given value of "traditional" anyway, but yeah, did my masters in a traditional archives program 'n all. [03:04] mistym: Spiffy; just got mine [03:05] Congrats! [03:05] Thanks :) [03:05] Are you with a place now? [03:05] Yeah, I work at a museum in Manitoba. [03:06] Gotcha; bit distant for MARAC, then [03:07] Yeah, not exactly in the area. [03:07] Is there a regional one for central Canada? [03:08] Not in my province, at least. Alberta's archivists are pretty active though. [03:08] Oh, wow; that's a hike [03:11] The digital object seminar was cool; the job one, harrowing; the DH one, dull [03:12] I saw anarchivist tweeting about the job one. It sounded brutal. [03:12] Yep [03:13] Lots of "the system is totally fucking broken and we can't fix it" [03:14] And more of the usual student/professional divide, but nobody discussed it :( [03:14] :( [03:15] For those of us stuck between them, we're SOL [03:18] Hm, is there a url for the wayback machine to load the latest version of a page, rather than a list or specific revision? [03:24] But yeah, after listening to a bunch of "real" archivist talk digital records for a day, I appreciate AT that much more [03:26] shaqfu: I know, right? It's unfortunate that it's hard to have a constructive discussion in that environment. [03:27] mistym: Yeah, lots of "we need to be doing something!" and nothing getting done [03:27] Yeah... [03:27] Aah, I was just about to ask about that... [03:28] shaqfu: Giving a talk at this year's ACA that I'm hoping to balance with a little "we can do things! let's get things done!" [03:28] mistym: C being Canadian or Certified? [03:28] Canadian. [03:28] ("Canuck") [03:29] Wyatt|Wor: re: wayback machine? [03:29] If a bunch of malcontents online can move mountains, imagine how much big institutions could do... [03:29] mistym: Re: "listening to a bunch of "real" archivist talk digital records for a day" [03:29] Wyatt|Wor: Ahh. [03:29] Whoops, those quotes escaped. [03:30] Need to be more careful escaping. [03:31] shaqfu: I dunno, the longer I'm in big institutions, the more I worry institutional glacial workflows can't be made to work at the speed that's useful. [03:31] that archive team was necessary shows that there's a real problem with "real" archivists [03:31] I'm being overly pessimistic there, but there is major reorientation that institutions, in the big-institution sense, are going to have to do. [03:31] as much as many of them do great work, a lot of them have a lot of great ideas about digital preservation and very little wget [03:32] winr4r: I think there are more problems than simply archivists. [03:32] Wyatt|Wor: oh of course [03:32] i don't doubt that [03:32] mistym: Yeah, it's hard to respond to "you have 30 days before we delete everything" at the speed of bureaucracy [03:32] Mhm... [03:33] yes [03:33] It's also hard to change the mentality of people who put their data up without a second thought. [03:33] Yeah :( [03:33] And harder still to change the businesses that will bean-count something into oblivion without so much as a half-hearted apology. [03:33] And yeah, there needs to be a serious realignment if we're going to realistically deal with these records [03:34] Christ Almighty, Geocities alone is bigger than most university library systems [03:34] Good fucking luck doing it the old-fashioned way; I'll see you at the heat death of the universe [03:34] And yes, the metadata problem is monstrous. [03:34] Wyatt|Wor: It's solvable - you can work miracles with machine language processing [03:35] Which can at least deal with text stuff [03:35] give it a decade or so [03:35] shaqfu: Haha, you got me. I was just thinking about how to hack away at it with NLP. [03:35] Wyatt|Wor: One panel today was about using topic modeling on newspapers; I'm sure, given time, it'll apply to messier collections [03:36] But yeah, get a lot of processing power together, point it at Geocities, and you'll have something at least usable [03:36] maybe the only way to handle the new workflows is to have a totally separate group inside that's connected in name only, so they can respond in the timeframes required [03:37] shaqfu: I see Yahoo as being a pretty good analogue. Not data deletion Yahoo, but the old Yahoo web directory. [03:37] Ah, yeah [03:37] I've heard (unconfirmed) that they were among the biggest employers of library-school graduates at one point in time! [03:38] Yeah, like DMOZ, except people used it [03:38] Exactly. [03:38] There was a point where people thought of the internet as a thing you could index by hand, with meticulous metadata. [03:38] Then the dot-com boom came, the internet *exploded*, and that was never possible again. [03:38] Yep [03:39] mistym: i still think there's a place for it [03:39] winr4r: For hand curation? [03:39] shaqfu: yes [03:39] actually we have it already: it's called twitter [03:39] winr4r: Possible, but things move too fast for that [03:39] you just distribute the task [03:40] winr4r: For subject-specific stuff, etc. What I'm saying is that there will never again be a time where all the Internet that's fit to print is hand-curated to someone's professional standards. [03:40] winr4r: That requires an established network [03:40] shaqfu: yes [03:40] I think a better example is any of the social bookmarking sites (or any bookmarking site that has bookmarks open to the public) [03:40] like delicious and pinboard [03:41] dashcloud: Not enough. There's simply too much data to rely on that. [03:41] mistym: you might be right [03:41] on the other hand, is there actually more good stuff on the internet than there was in 1999? :P [03:41] hell yes [03:42] in quantity yes, percentage wise maybe maybenot [03:42] HiiiiIiIiiIIiiiI [03:42] I saw "Detention" [03:42] you must see detention. [03:43] Detention? [03:43] But I've done nothing wrong! [03:43] hi jason [03:43] Movie about chewing gum in high school? [03:43] Evening to you. [03:45] SketchCow: Has there been talk at IA about using NLP to handle metadata for these McLargeHuge collections? [03:47] What does NLP = ? [03:48] Quick search shows Neuro Linguistic Programming, but that doesn't seem right. [03:48] That's probably it. [03:48] natural language processing [03:48] If you're trying to data mine metadata for relevant info to humans [03:48] winr4r: Thanks [03:51] Letting machines figure out word associations, more or less [03:51] Interesting. [03:52] I'm reading the The Stanford Natural Language Processing Group web page. Who knew? Well I guess you did. :) [03:53] shaqfu: Absolutely no [03:54] SketchCow: Really? Admittedly, I'm surprised [03:55] Seems like the only reasonable solution - barring some miracle, I don't see there being enough humans to mark everything up [03:57] Don't be the latest in a hundred people I've dealt with surprised that archive.org doesn't have much manpower. [03:58] SketchCow: I'm not surprised IA has minimal staff; I knew that already :P I'm surprise there's been no talk about letting machines do the heavy markup lifting [03:59] It's kind of easy to assume that archive.org is some all-powerful automoton. Then you stand in a room WITH THE INTERNET and suddenly you realize that it's not actually powered by elder gods or smth. [03:59] (even if it *is* in a church) [04:01] Again [04:01] THERE'S NOBODY TO TALK [04:01] i'd say let someone in 20 or 30 years deal with it when they're widely recognised for being as important as they are [04:01] SketchCow: Gotcha this time [04:01] you could worry about NLP now to find all those cat photos, or you could 'grep -ri "cat\.*photo" /geocities' a thousand times as fast in 20 years' time [04:02] winr4r: For the amount of processing power it'd take, and how NLP isn't really at the point you'd need yet, yeah, may as well wait [04:03] I don't think there's only one "right" approach. [04:03] There rarely is [04:03] Especially since the data is hierarchical that actually could help quite a bit. [04:04] Well, up to a point. [04:05] (I'm not familiar enough with the data set to know how what proportion of neighbourhoods are just numbered with random stuff shoved in) [04:05] Didn't they stop that system after a point? [04:05] shaqfu: yes, after around 1999 i believe [04:06] So it's hard to regard that hierarchy for any serious use, unless you're limiting your work to 199x-1999 [04:06] One harpoon. [04:06] Hm? [04:07] Hooray, scraping script running. Hopefully will be successful! [04:07] mistym: what are you scraping? [04:07] winr4r: digiplay.info [04:07] mistym: excellent [04:08] Even though the data is kind of messy, it's not too much work to extract it into structured json. [04:08] This is also why I keep bringing in assholes from outside to dump open-source solutions and leverage archive.org against it [04:09] There's no dev space inside the company [04:09] leveraged synergies [04:09] wtf [04:09] SketchCow is talking about leveraged synergies [04:09] Hunh; I knew it ran lean, but didn't expect it to run *that* lean [04:10] SketchCow Clicker? [04:10] The press says things like 200-300 employees [04:10] But the vast vast vast majority of those people are scanners. Book scanners. [04:10] Isn't it something like 20-30 core? [04:10] I'd say, maybe, MAYBE, my observaton is 20-30. [04:10] Yes, 20, 30. [04:11] Now, work that out. [04:11] We have 5 people overseeing the book scanning centers. [04:11] Boom, now we're 20-25% down [04:11] etc [04:12] I'm like hiring six new employees in terms of stuff and publicity and the rest [04:12] But I can only do things that are being brought in, there's no way to make those poor devs do MORE work [04:12] 15 coder-librians is not enough [04:12] And there we are. [04:12] And those people are also jointly responsible for running all the servers and such? [04:12] So if we do some sort of NLP smart tagging smartiness, great. Get on it. [04:12] Free tour. [04:12] Yes, there's a team of 5-10 dev/admin/network people [04:13] OH LOOK AT ALL YOUR EYES GO WIDE [04:13] Anyway, so yeah, get on it. [04:13] and there's got to be like six billion servers there [04:13] I'll just use my universal access to ensure you get stuff to help you. [04:14] you should get a rifle, some tranq darts, then go hang out outside one of google's datacenters [04:14] Pity 'bout the 5-10 years math ed it'd take to do it; it'd be a badass project [04:14] Go rape a gaduate program is my suggestion [04:15] Anyway, unrelated, I need to go to bed now. [04:15] Yeah, the perfect admin abduction is hard to pull off. [04:15] Good night. [04:15] night jason [04:15] G'nite [04:15] Let's keep making amazing shit [04:17] Ooo, one of batcave's two remaining mounted drive sets has been emptied out [04:17] We're now down to one. 9gb. [04:17] in any case, i think there's a risk of over-complicating the "saving shit" strategy (which provably works very well) and turning it into a discussion about "how do we make sure that every single thing is categorised as well as books are in a library" and thereby getting very little done [04:17] SketchCow: s/g/t/ ? [04:17] i can't imagine you having only 9gb of *anything* [04:17] ha ha [04:18] winr4r: Yep, that's what happened on this end; it turned into an issue of description [04:18] Did I write 9gb? [04:18] I DO need a rest [04:18] 9tb [04:18] Which, really, people only care 'bout good-enough [04:18] SketchCow: did you actually sleep at all last night? [04:18] Barring special cases - obviously shit like presidential letters need Awesome [04:19] meh, president is just another sack of meat [04:19] shaqfu: good enough and actually existing beats immaculately described archives that do not [04:19] winr4r: You got it [04:20] "less process more product" etc [04:20] mistym: wrought grand - okay item-level description [04:21] ~500 pages of 5000. This may take awhile. [04:24] good luck [04:30] So as a baseline, any thoughts on what metadata should be given priority? Dublin Core and a mostly-flat ontology of descriptive tags? [04:34] http://archive.org/details/stage6 [04:35] He even archives in his sleep. ;) [04:36] zzzzzzreclassifyfuckdublincorzzzzzzmmzzzzz [04:39] haha [04:42] That's fine. I'm not a particularly huge fan of DCMI, even though I live a stone's throw from Dublin. [04:42] mmm [04:42] i has a collection [04:43] What has you a collection of? [04:43] i'm uploading the stage6 items [04:43] Coderjoe: how did you get them? [04:44] Ahh. [04:44] winr4r: I downloaded them between the closing announcement and the shutdown [04:44] with metadata and everythign [04:44] http://wegetsignal.org/stage6/ [04:44] will probably be a little slow [04:45] 25 terabytes? [04:45] <3 [04:45] winr4r: I don't have 25 TB of videos [04:45] only 290-ish GB [04:45] oh, nm, saw the percentage [04:45] but still, good work :) [04:46] Good one. [04:46] Is that 25TB before or after deriving? [04:46] that was the projected size of what was up on the stage6 servers [04:47] Ah... :/ [04:48] on an unrelated note, is there a big list of fortunecity sites that you guys have been using? [04:48] and this was just me with three network connections (home, work, and a server in california) [04:48] if i'm well within my bandwidth cap towards the end of the month, i will set my screenshot bot loose again [04:48] winr4r: I think it came from google results [04:49] http://archive.org/details/geocities-screengrabs-collection in case you didn't know [04:49] 4000+ from geocities [04:50] winr4r: Does it run as a normal user? I give you an account on my VPS, if you'd like. [04:51] Wyatt|Wor: yes, though it does need an xvfb to run on [04:51] and a bunch of dependencies that aren't normally on a server [04:51] That doesn't necessarily mean it's not doable. [04:52] (Though I've never messed with xvfb on a headless machine) [04:52] Wyatt|Wor: me neither [04:55] xvncserver wouldn't work? [04:55] (you should be able to take a shot of the desktop or the like, I would think) [04:55] Coderjoe: i'd expect it would [04:55] i just know for sure that Xvfb does [04:56] i mean, yes, it isn't a vfb, but still [04:56] hmm. 6 items i need to redo [04:57] i'm sure it will increase [04:57] Okay, looks like xvfb will work on a headless box. That's what Google says. [04:58] btw, that "videos listed" stat is just the videos I had pulled into my database with my importer. the "total video count" at the bottom is my estimated total video count that stage6 hosted [04:58] Wyatt|Wor: splendid! [06:57] underscor, what dump server? [06:59] ah, your.org [09:15] Curious, since this thing is _still_ grepping that file, the webdav-feed.json and .xml...what role do they serve, exactly? [09:15] how big is the file? [09:16] and how can it take that long to grep anything? [09:16] fgrep is much faster, for fixed strings [09:17] yeah but i grepped a 1.9gb file in seconds, earlier today [09:17] it was probably all sitting in ram [09:17] (getting a list of fortunecity sites from the ODP) [09:17] chronomex: nope, fresh from the disk [09:17] hm, ok [09:19] It's the json is 35MB. The incantation is grep http://gallery.me.com/[^"<]+ data/p/pe/per/pertormod1/gallery.me.com/webdav-feed.json and it's accumulated 3766 CPU _Minutes_ [09:19] okay, i would call that a bug [09:20] indeed [09:20] Run it interactively and see what it outputs, if anything? [09:20] use [^"<]+? and grep -E [09:20] Even assuming the worst case of grep's iconv locale performance, I'm inclined to agree. [09:20] Sorry, there's an -oE in there I missed [09:21] (It's the seesaw-s3.sh) [09:21] just timed a regex again on a copy of said 19gb file, 19.4 seconds [09:21] winr4r: 1.9 or 19? [09:21] 1.9gb* [09:21] +? [09:21] ah [09:21] +? [09:21] so yeah 3766 minutes for a 35mb file is a LITTLE excessive [09:22] Wyatt|Wor: hmmmm. I would try some cut | grep(not -E) action. [09:22] That comes out to about 162 bytes per second (and dropping, if it's still going) [09:22] drooping [09:23] That's my perspective too. I think the emulated ARM processor that booted Linux on the 8-bit MC had a better data rate about a thousand times that. [09:25] winr4r: I see you're asking about a list of FortuneCity sites. I can send you the list from which we've been downloading, if that helps. [09:25] Hey, it's faster than some 600 baud modems, according to Wikipedia. [09:25] some? [09:25] alard: i'd appreciate it, we can compare notes too [09:25] https://en.wikipedia.org/wiki/List_of_device_bandwidths#Modems_.E2.80.93_narrow_and_broadband [09:26] alard: i grabbed a list from ODP (hence greeping a 1.9gb file), did you guys try that? [09:26] There are two 600 baud ones that're 1.2 kbit/s and one that's 2.4 kbit/s. [09:26] ah [09:26] winr4r: What's ODP? [09:26] alard: open directory project [09:26] Ah, I see. No, I just googled. [09:26] okay, one sec [09:27] http://dl.dropbox.com/u/57276499/sitelist.txt [09:28] is what i got from ODP [09:28] I actually don't understand this regex, even. o is --only-matching -E is extended regex... how does this work? [^"<]+ [09:29] I don't think you should need -E for that [09:29] win4r: Okay, got it. I'm currently making my list. [09:29] Wyatt|Wor: The regex matches anything until " or < [09:29] I believe that's matching urls in the webdav file. [09:30] So it will match from http:// until the tag ends. [09:30] alard: Ah, I thought the caret was an anchor to the beginning? [09:31] Wyatt|Wor: not within []s [09:31] Between [] it's a negation. So 'anything but " and < ' [09:31] alard: thanks :) [09:31] ooooooh, I see. Hmm, need to put more skill points in RegEx. And the +? [09:31] one or more instances of the preceding object [09:32] * means zero or more. [09:32] er, matching element, which in this case is the whole [] expression [09:32] Ah, so [] create a single semantic unit. I see. [09:32] indeed [09:33] it matches exactly one character [09:33] winr4r: http://db.tt/PjVwK1A2 (a 3.7MB .txt.bzip2) [09:34] is that the file we're working on? [09:34] No, that's the list of all fortunecity sites. [09:34] alard: thanks! [09:35] winr4r: You'll have to expand the streets yourself, we've basically archived anything from number 0 to 2600. [09:37] alard: so com/campus/athena = campus.fortunecity.com/athena/ ? [09:41] Hm, so it's definitely finding things, though it seems awfully slow... [09:42] paste the entire command line [09:43] grep -oE 'http://gallery.me.com/[^"<]+' data/p/pe/per/pertormod1/gallery.me.com/webdav-feed.json # Pretty much verbatim from seesaw-s3.sh [09:44] Ah, I think I've got the problem. Who do I bug about a patch? [09:45] a bug in grep? [09:45] Well, yes, to an extent. But it's a bug I think we can safely work around. export LANG=C And it's about three orders of magnitude faster [09:46] I could have sworn the iconv bug was fixed though. :/ [09:46] where is the file to parse? i want to make some grep tests [09:46] emijrp: Let me stick it somewhere. [09:48] Come to think of it , DCC would have been faster... [09:48] radiusic.com/bigfeet.json [09:48] "d" and "t" are totally right next to each other. [09:50] downlaiding [09:52] Okay yeah, it hit me because my grep is old. It's apparently fixed in grep 2.9 [09:53] (Didn't realise I was still using grep 2.5.4) [09:53] what do you want, the entire url o just the domain + username?= [09:53] grep 2.old [09:53] The problem is, in this case, most distros in production are probably using old grep. [09:54] CentOS 6 has grep 2.6 [09:54] i'm on 2.5.4 too [09:55] but piping it to a file, it takes a few seconds [09:56] 8 seconds, to be precise [09:56] time grep -oE 'http://gallery.me.com/[^"<]+' bigfeet.json > what [09:56] is what i am using [09:57] emijrp: The problem isn't that it doesn't work. The problem is that when you're using many versions of grep in the wild with LANG=en_US.utf8 (or any unicode, locale for that matter), it's fantastically slow. [09:59] unicode comparisons are always very slow [09:59] The good thing is, we can patch our scripts by explicitly setting LANG=C and LC_CTYPE=C and that should be safe. [10:00] i am en_GB.utf-8 and that grep still takes seconds rather than hours [10:00] (Or just unset LC_CTYPE) [10:01] In what format is the file you're grepping save in? Wouldn't this matter? [10:01] I suppose grep should determine what charset it's being used, but will do so only from the headers... [10:02] Nemo_bis: It's just a JSON file from mobileme [10:03] winr4r: "so com/campus/athena = campus.fortunecity.com/athena/ ?" Yes, or www.fortunecity.com/campus/athena/. (The subdomain approach doesn't work with co.uk/it/es/se, I think.) [10:03] alard: gotcha [10:48] netsplit [11:08] Okay, updated grep and things are much speedier. I'll try to take a look at the memac scripts when I get home and figure out where to add that env. [11:09] it's weird though [11:10] that i can be running the same version also with a UTF-8 LANG and do in nineteen seconds what your grep didn't finish in hours [11:10] * winr4r isn't exactly on a speed-demon computer [11:14] winr4r: What distro/version? [11:15] Wyatt|Wor: ubuntu 10.04 [11:15] Distro-specific patches will do that. Yeah, Debian patched it a while back. [11:15] ah :) [11:15] Gentoo just stabled a newer version instead. [11:15] that explains that [11:16] (But this is my work computer, so I don't exactly bother updating often) [11:17] * winr4r nods [11:17] 1507 screenshots ;D [11:18] Ooh, going pretty fast. [11:27] 06:25:13 up 13:46, 6 users, load average: 54.70, 54.97, 58.34 [11:27] hmm i think i started too many threads [11:27] What, that's it? [11:27] hahaha [11:27] yeah that's it [11:28] ;) [11:30] i got a box from softlayer and its not going over 100mbit :( [11:31] Time to go home. Later. [11:32] bye Wyatt|Wor! [13:33] See? I sleep like everyone else. Here I am, back up again. [13:34] lies [13:40] haha [13:41] hey cow, i'll have fortunecity screenshots for you soon [13:41]