[09:33] <emijrp> im going to create profiles for a fuck ton of wikis, in wikiindex
[11:19] <emijrp> http://wikiindex.org/index.php?title=Special:NewPages&limit=100
[11:21] <emijrp> http://awa.wikkii.com/wiki/Anti-Wikia_Alliance ha
[13:00] <Nemo_bis> hmpf
[15:22] <sethish> emijrp: I've been meaning to poke you about wikis
[15:23] <emijrp> tell me
[15:23] <sethish> emijrp: I have some code that uses pywikipediabot to archive a wiki via the mediawiki api.
[15:23] <sethish> In my experience, nearly every wiki I have checked has left their api on.
[15:24] <sethish> And this way it lets me get the wikimarkup text and/or mediawiki's rendered html.
[15:25] <emijrp> dumpgenerator.py uses api to get the page titles, and special:export to export all
[15:26] <sethish> What format does it get them in?
[15:26] <emijrp> xml
[15:26] <sethish> I remember looking at the wikiteam repo, but it has been a few weeks (months?)
[15:26] <sethish> Hrrm.
[15:27] <sethish> Where are wikis being archived/saved?
[15:27] <sethish> Is there a single location?
[15:27] <emijrp> no
[15:27] <emijrp> internet archive, google code, and private servers
[15:29] <emijrp> dumpgenerator.py works well in most mediawikis (some old versions or servers are buggy, but it is 1%)
[15:29] <emijrp> now, we want to generate a list with all internet mediawikis, and ragedownload all
[15:30] <sethish> mmm. That would be ideal, but many wikis are not under any kind of open license.
[15:30] <sethish> Does the xml dump include history?
[15:30] <emijrp> sure, the whole history
[15:30] <sethish> Nice
[15:31] <sethish> Forgive me if I wax philosophic for a minute, I have been thinking about archiving as a commitment lately.
[15:32] <sethish> That if I scrape something and store it elsewhere, it is my responsibility to maintain that archive.
[15:33] <emijrp> we try to upload all to internet archive
[15:34] <emijrp> that guys are good at archiving stuff
[15:34] <sethish> Sure, but there are other responsibilities involved, archive.org will keep things around for sure.
[15:34] <sethish> But going back every N months and getting a fresh dump.  Or keeping an eye out for fork of a given wiki and getting an archive of them.
[15:35] <sethish> I happened to have a copy of the encyclopedia dramatica wiki when the O' Internet thing happened
[15:36] <sethish> and I felt that it was my responsibility to publicize the existance of the archive to folks who wanted to fork
[20:03] <chronomex> sethish: <3
[20:13] <sethish> ?
[20:18] <chronomex> 08:35:15 < sethish> I happened to have a copy of the encyclopedia dramatica wiki when the O' Internet thing happened
[20:22] <sethish> Oh, yeah. I am not cited on wikipedia tho :-[ and I can show a direct correlation to my dropping the dox and them having content
[20:22] <sethish> but oh well.
[20:26] <emijrp> you did it for teh lulz
[20:31] <emijrp> 2011-04-05: First commit.
[20:32] <emijrp> http://code.google.com/p/wikiteam/wiki/News
[20:33] <emijrp> Happy Birthday WikiTeam.
[20:40] <chronomex> funny, I just got a nastygram from the bing maps api robot saying that I've exceeded some sort of unknown account limit
[20:40] <chronomex> which appears to be the "thanks for using our api for the last year, now fuck off!"
[23:57] <Nemo_bis> chronomex, probably that's because everyone's using OSM anyway nowadays
[23:57] <chronomex> probably
[23:57] <chronomex> I'll just have to transition over, I guess ...