[06:15] <Nemo_bis> FYI, I'm getting about 1100 new dumps per day on dumps.wikia.net
[06:16] <Nemo_bis> If it goes on like this, in 32 weeks we're done O_o
[06:26] <cmx> awesome!
[06:26] <cmx> done with wikia, that is :P
[06:27] <omf_> and I am going to download all those when they get up on IA ;)
[06:44] <Nemo_bis> omf_: why don't you instead help me by downloading them all and uploading to IA? :)
[06:44] <Nemo_bis> like http://archive.org/details/wikia_dump_20121204
[06:44] <Nemo_bis> it's just recursive wget and some zipping :)
[06:46] <omf_> I can help a little this month and more next month. See I already blew my budget for the month on screenshots of posterous and only got so far
[06:46] <omf_> I am trying to keep costs down since I used to rack them up pretty high
[06:47] <omf_> My home internet is shit
[06:48] <omf_> I tried the higher speeds before but the quality of service was shittier so I downgraded.
[06:48] <Nemo_bis> Ah, not best person to ask then. :)
[06:48] <omf_> Posterous was different
[06:49] <omf_> that required a lot of CPU power so it cost more
[06:49] <Nemo_bis> I can do it, I just have to format this stupid external sata hdd
[06:49] <omf_> how big a chunk are we talking about?
[06:49] <omf_> size wise
[06:49] <Nemo_bis> no chunks
[06:49] <Nemo_bis> just a wget of some 3-400 GB
[06:50] <Nemo_bis> I can do it myself, no worries
[06:50] <Nemo_bis> aww underscor is not even in this channel any longer, traitor :|
[10:28] <Nemo_bis> as soon as fix this I'll archive the Wikia dumps http://p.defau.lt/?eeBaa4Jb1zuYi0JR0DPZYg
[16:19] <underscor> Nemo_bis: I'm not?
[16:19] <underscor> SketchCow: ops spread
[16:20] * Nemo_bis faceplams
[16:20] <Nemo_bis> (twice for typos)
[16:20] <Nemo_bis> underscor: so can you do a wget -r of dumps.wikia.net and upload the zips to archive.org? :D
[16:22] <underscor> I suppose ;p
[16:22] <underscor> Do these include the media?
[16:46] <Nemo_bis> underscor: they should but it's not clear to me at what conditions they are created
[16:46] <Nemo_bis> I guess most wikis just don't have any image
[16:47] <Nemo_bis> Last times I just made a .zip for each letter https://archive.org/details/wikia_dump_20121204
[16:51] <omf_> I too have noticed many of the wikia community sites are seriously lacking images, I always put it down to not having enough volunteers
[16:52] <omf_> just getting a good skeleton of data into place takes a lot of effort
[16:57] <Nemo_bis> well, or they just use Wikimedia Commons
[17:00] <omf_> wikimedia commons has pretty harsh standards for inclusion, that doesn't work on content sites like Star Trek where there is no CC content unless it was fan generated at a conference
[17:00] <omf_> most wikia sites that have images do not own the images at least from what I have observed of the Scifi wikia sites
[17:48] <SketchCow> WHAT DID YOU ALL DO
[17:51] <cmx> I did nothing wrong!
[17:56] <sethish> :-|
[18:52] <underscor> I blame poor training by our leader
[19:11] <sethish> underscor, have you backed up any semantic mediawiki wikis?  Is there anything special one needs to do?
[19:12] <underscor> semantic?
[19:12] <underscor> I'm not familiar :c
[19:15] <Nemo_bis> sethish: afaik, no
[19:15] <Nemo_bis> everything comes from the wiki pages and can be rebuilt from there
[19:16] <Nemo_bis> the question is rather what it takes to *import* a semantic wiki
[19:53] <sethish> mmm, how are y'all generating dumps these days?  Scrape script, or do you have a mediawiki-api wrapper to dump wikimarkup?
[19:53] <sethish> Both are important, scrape always works
[19:54] <sethish> underscor, I'm also isforinsects, we've spoken before.  I did the backup of Encyclopedia Dramatica a few years ago that ended up getting used to rebuild the site
[19:58] <Nemo_bis> we use dumpgenerator.py
[19:58] <Nemo_bis> we'd like to use the API more
[20:00] <sethish> Have you ever done any mass upload scripts?  I have a big collection of images from the CDC that I need to get over to wiki commons and I would love help
[20:02] <cmx> cult of dead cow or center for disease control?
[20:02] <cmx> :P
[20:08] <sethish> Center for Disease Control (and Prevention)
[20:08] <sethish> It's the Public Health Image Library
[20:08] <sethish> I scraped it ages ago
[20:08] <sethish> With good metadata
[20:08] <cmx> nice
[20:13] <omf_> sethish, I have mass upload scripts for the Internet Archive using their s3 api if you want to upload it there too :)
[20:27] <Nemo_bis> sethish: for Commons we have several tools
[20:27] <Nemo_bis> there's the classic uploader.py
[20:28] <Nemo_bis> https://outreach.wikimedia.org/wiki/GLAM/Resources/Tools
[20:29] <Nemo_bis> https://commons.wikimedia.org/wiki/Commons:Batch_uploading#Tools
[20:30] <Nemo_bis> then I think ingester.py or so on PWB
[20:31] <cmx> ingester.py makes me think of a chipper-shredder
[20:35] <omf_> Didn't we need a Encyclopedia Dramatica grab?
[20:36] <Nemo_bis> did we?
[20:36] <Nemo_bis> I made one some time ago iirc
[20:57] <omf_> I just remember being asked about it more than once
[20:57] <omf_> how old is your backup Nemo_bis
[21:00] <Nemo_bis> just search? :)
[21:02] <Nemo_bis> https://archive.org/details/wiki-encyclopediadramatica.ch
[21:03] <Nemo_bis> Publicdate: 2012-02-29 08:38:47
[21:03] <Nemo_bis> should be this
[21:10] <sethish> My ED dump is from 2011
[21:11] <sethish> ED.ch posts links to several more recent dumps
[21:29] <Nemo_bis> upload them then
[21:32] <omf_> Is this their new official home? http://dramatica.in/
[21:37] <Nemo_bis> no idea
[21:45] <omf_> That url before was just a landing page for https://encyclopediadramatica.se/Main_Page
[21:50] <sethish> I think encyclopediadramatica.ch is the canonical url
[22:01] <omf_> OpenDNS says that encyclopediadramatica.ch does not exist
[22:01] <omf_> hence why I am poking around at it
[22:08] <omf_> ED is starting to move TLDs as often as TPB
[22:37] <sethish> Oh, I had it backwards
[22:38] <sethish> it _used_ to be .ch, and moved to .se
[22:38] <sethish> sorry