| Time |
Nickname |
Message |
|
00:28
🔗
|
SketchCow |
Lord_Nigh: Thanks |
|
01:43
🔗
|
BlueMax |
I want to start a project to archive Doom-related user generated content |
|
01:51
🔗
|
BlueMax |
I'm just not sure where to start |
|
02:10
🔗
|
odie5533 |
BlueMax: Sounds like a fun project. |
|
02:10
🔗
|
BlueMax |
Mmm. |
|
02:12
🔗
|
odie5533 |
might be a good idea to have a portal to showcase the archived items. |
|
02:21
🔗
|
BlueMax |
a portal like on archive.org? |
|
02:23
🔗
|
odie5533 |
Maybe. Up to you I suppose. |
|
02:25
🔗
|
BlueMax |
see what I mean by not knowing where to start :P |
|
02:26
🔗
|
odie5533 |
well, how about starting by just writing down items you'd archive |
|
02:26
🔗
|
odie5533 |
perhaps websites, maps, movies, graphics. |
|
02:26
🔗
|
BlueMax |
like a mission statement of some kind. |
|
02:27
🔗
|
odie5533 |
sure, that as well. |
|
02:27
🔗
|
BlueMax |
because Doom content takes many forms |
|
02:32
🔗
|
BlueMax |
The *** project is dedicated to the archival of user-generated content based around Doom and games based on the "Doom Engine". This includes original game versions and source ports, tools for modifiying the base game and WADs, user generated WADs and PK3s, websites and discussion forums based around the Doom engine, and any other forms of content. |
|
02:32
🔗
|
BlueMax |
man I feel dirty just writing that for some reason |
|
02:37
🔗
|
odie5533 |
dirty? why |
|
02:38
🔗
|
BlueMax |
no idea. |
|
02:38
🔗
|
BlueMax |
all businesslike. |
|
02:42
🔗
|
BlueMax |
But I think if I wanted to start anywhere it would be backups of the sources WadSeeker and similar tools use |
|
02:42
🔗
|
BlueMax |
like the /idgames FTP |
|
02:49
🔗
|
odie5533 |
BlueMax: http://searchworks.stanford.edu/view/rd383mp8260 http://purl.stanford.edu/rd383mp8260 |
|
02:51
🔗
|
BlueMax |
well if that's not a good start I dunno what is. |
|
02:53
🔗
|
odie5533 |
Their mission is a bit different though. So keep that in mind |
|
02:54
🔗
|
BlueMax |
Yeah, it appears they don't want much to do with saving the user made content |
|
03:03
🔗
|
BlueMax |
is anyone else interested in helping with this project? I'm not going to be able to do it all myself...I might be able to if I had internet that could upload faster than 100KBps :| |
|
03:06
🔗
|
odie5533 |
I could perhaps try to help, though I'd rather not commit to anything specific. |
|
03:06
🔗
|
odie5533 |
100 Kbps is more than enough to tell a remote server to upload and download something. |
|
03:08
🔗
|
BlueMax |
I don't have money for a remote server |
|
03:08
🔗
|
odie5533 |
I think there are lots of people that run warriors who are willing to, in effect, be the remote server for you. |
|
03:10
🔗
|
BlueMax |
The Warrior serves a specific purpose which I don't think a project like this counts for |
|
03:11
🔗
|
odie5533 |
also, Amazon EC2 offers a 1 year free trial |
|
03:13
🔗
|
BlueMax |
I'm not sure I understand what EC2 actually is |
|
03:14
🔗
|
odie5533 |
remote server. |
|
03:15
🔗
|
odie5533 |
Is it mostly archiving FTPs? |
|
03:16
🔗
|
BlueMax |
yeah, and specific websites like Doomworld |
|
03:16
🔗
|
odie5533 |
I am not sure how you'd package the FTP, but I'm sure someone here knows how that's done |
|
03:16
🔗
|
odie5533 |
with websites you could use a warrior script, or archivebot (w/ permission), or running wget on e.g. an EC2 instance. |
|
03:17
🔗
|
BlueMax |
there's a couple of sites that aren't exactly opening their contents to the public that I want to contact to get their files |
|
03:19
🔗
|
BlueMax |
but yeah a good start would be grabbing a backup of idgames ftp and putting it somewhere safe |
|
03:52
🔗
|
Lord_Nigh |
another good place to look might ironically be asking john carmack if he has any pre-release stuff he can share |
|
03:53
🔗
|
Lord_Nigh |
and if not for copyright reasons it can be blacked out (though theres lawyers probably who will prevent that) |
|
03:53
🔗
|
Lord_Nigh |
annoying lawyerses |
|
03:53
🔗
|
BlueMax |
you think so? I thought he would've passed that threshold when he released the Doom source code back in the late 90s |
|
03:53
🔗
|
Lord_Nigh |
nah i meant wads which went with the doom betas and stuff |
|
03:53
🔗
|
Lord_Nigh |
which technically may contain non-shareware assets |
|
03:55
🔗
|
BlueMax |
ah I see |
|
03:56
🔗
|
BlueMax |
That should be a goal for later |
|
03:58
🔗
|
Lord_Nigh |
i'd actually really like to see if bobby prince still has the midi/data files used to create the dumbed down yamaha music from commander keen |
|
03:58
🔗
|
Lord_Nigh |
shown in the video at http://www.youtube.com/watch?v=9w3yoIOK-9U |
|
03:58
🔗
|
Lord_Nigh |
from 1991 |
|
03:59
🔗
|
DFJustin |
I'm pretty sure all the doom prototype stuff that is available has been released already |
|
04:00
🔗
|
Lord_Nigh |
DFJustin: possible, but can't hurt to ask |
|
04:00
🔗
|
Lord_Nigh |
http://bobbyprincemusic.blogspot.com/ |
|
04:00
🔗
|
DFJustin |
http://doomwiki.org/wiki/Versions_of_Doom_and_Doom_II |
|
04:01
🔗
|
Lord_Nigh |
that note in the latest post on bobby prince's blog implies valve may have a policy in place now that game soundtracks may not be sold outside of steam if offered on steam at all? |
|
04:01
🔗
|
Lord_Nigh |
is that true? |
|
04:01
🔗
|
BlueMax |
well I can see why that would be given Steam's want to release soundtracks as seperate DLC |
|
04:01
🔗
|
DFJustin |
that would be odd if true because they let you sell the game on the side |
|
04:02
🔗
|
Lord_Nigh |
i guess it doesn't affect gaslamp games which never sold the dredmor soundtrack on steam, they always had it available for free (except the flac version) on bandcamp |
|
04:03
🔗
|
BlueMax |
maybe we need a dedicated channel for this talk soon |
|
04:03
🔗
|
DFJustin |
godane: I remember d-addicts, used them to watch Densha Otoko |
|
04:03
🔗
|
DFJustin |
they seem to have good staying power but the more archiving the better |
|
04:05
🔗
|
phillipsj |
DRM companies always demand exclusive content: they are the publishers, not the er, "publishers." |
|
04:05
🔗
|
DFJustin |
re: doom I would think mirroring /idgames would get you the vast majority of material, except for weird one-off stuff on geocities type pages of people who weren't plugged into the doom community |
|
04:06
🔗
|
phillipsj |
idgames are just the shareware files, AFAIK. |
|
04:06
🔗
|
odie5533 |
isnt there new stuff be created though? |
|
04:06
🔗
|
BlueMax |
there is plenty of stuff not hosted on idgames |
|
04:06
🔗
|
DFJustin |
we should have pretty good coverage of wads 1994-1996 in the shareware cd archive but the later stuff would be great to save |
|
04:06
🔗
|
BlueMax |
like a ton of the multiplayer focused content |
|
04:06
🔗
|
odie5533 |
yea |
|
04:07
🔗
|
BlueMax |
a lot of that can be gotten off places like best-ever |
|
04:07
🔗
|
BlueMax |
which was part of my plan to get |
|
04:07
🔗
|
BlueMax |
also DogSoft has about 10,000 MP focused wads but they don't mirror them in a way that makes it easy to get them all at once |
|
04:07
🔗
|
DFJustin |
phillipsj: no, it's user content too http://www.doomworld.com/idgames/ |
|
04:07
🔗
|
DFJustin |
only the idstuff subdirectory is the official shareware releases |
|
04:12
🔗
|
BlueMax |
should I make a dedicated channel for the doom talk? |
|
04:13
🔗
|
Lord_Nigh |
(i'm done with doom talk personally, dunno about everyone else) |
|
04:13
🔗
|
odie5533 |
prob get more input here, but if it gets to be too much for this chan then sure |
|
04:15
🔗
|
BlueMax |
OK I'll leave it for now then |
|
04:16
🔗
|
BlueMax |
I've just got a EC2 instance intitalizing |
|
04:19
🔗
|
BlueMax |
but yes Justin the first goal is indeed a backup of idgames |
|
04:19
🔗
|
BlueMax |
I'd like to get the doomworld idgames frontend as well but I'm not sure how to handle that |
|
04:33
🔗
|
BlueMax |
well this may not end well, either it's a Windows VM with a stupid small amount of HDD space and RAM or it's an Ubuntu CLI I'm not familiar with >___> |
|
04:34
🔗
|
Lord_Nigh |
http://bobbyprincemusic.blogspot.com/search?updated-min=2010-01-01T00:00:00-05:00&updated-max=2011-01-01T00:00:00-05:00&max-results=10 |
|
04:34
🔗
|
Lord_Nigh |
that first blog post pretty much explains how (but not why) the music industry is so utterly fucked up wrt copyright |
|
04:49
🔗
|
BlueMax |
so how much can you download on the free amazon instances before they start charging you |
|
04:51
🔗
|
BlueMax |
15GB bandwith huh...that's not enough to even get a copy of idgames :/ |
|
04:51
🔗
|
DFJustin |
give sketchcow an ftp url and he can suck it down to fos |
|
04:52
🔗
|
BlueMax |
I dont' want to keep bothering him for when I'm trying to suck down GetWAD servers |
|
07:37
🔗
|
odie5533 |
how big is GetWAD? |
|
07:38
🔗
|
odie5533 |
Also, if you don't have your own server, you will either need to do it with your home computer, or ask someone else to. Those are the only options. |
|
07:50
🔗
|
Lord_Nigh |
yeah this 50gb dump i ddi is gonna be the last one until january, i don't want comcast throttling the cable here, it has a 'soft' 300gb limit for each of up and down iirc |
|
07:50
🔗
|
Lord_Nigh |
per 3 months |
|
07:50
🔗
|
Lord_Nigh |
or so i think |
|
07:58
🔗
|
BlueMax |
odie5533, they're not relatively big downloads, the problem is uploading them, since I can't upload fast at all |
|
08:00
🔗
|
odie5533 |
If someone knows the easy/proper way to dump an FTP, I could run a small FTP dump script on my server. |
|
08:00
🔗
|
odie5533 |
and I assume other people here have servers that they might let you use for a good cause. |
|
08:01
🔗
|
BlueMax |
Maybe. |
|
08:01
🔗
|
BlueMax |
Also I'd like to back up the Doomworld frontend for idgames at the same time, but I'm not sure how I'd go about that |
|
08:02
🔗
|
odie5533 |
Is that a website? |
|
08:05
🔗
|
BlueMax |
Yeah, really simple one too |
|
08:05
🔗
|
BlueMax |
http://www.doomworld.com/idgames/ |
|
08:06
🔗
|
BlueMax |
mainly wanted to back it up for user reviews. |
|
08:07
🔗
|
SketchCow |
Doooom |
|
08:08
🔗
|
BlueMax |
hi Jason |
|
08:08
🔗
|
BlueMax |
what's up |
|
08:08
🔗
|
chfoo |
SketchCow: can you move and fix the mediatype for https://archive.org/details/dopplr_fire_grab_1 and https://archive.org/details/dopplr_fire_grab_2 |
|
08:11
🔗
|
SketchCow |
fire_1 was already fine, but 2 is now in there as well. |
|
08:12
🔗
|
chfoo |
SketchCow: sweet, thanks. |
|
08:28
🔗
|
odie5533 |
BlueMax: I'm not sure how well wget handles files served from external sites. |
|
08:28
🔗
|
BlueMax |
Here's the thing. |
|
08:28
🔗
|
BlueMax |
Doomworld just pulls from idgames. |
|
08:28
🔗
|
BlueMax |
So all we need to do is collect the webpages. |
|
08:28
🔗
|
odie5533 |
_All_ the downloads are already in the idgames ftp? |
|
08:29
🔗
|
BlueMax |
All of them. |
|
08:29
🔗
|
odie5533 |
where's the ftp for this one? http://www.doomworld.com/idgames/?id=17393 |
|
08:30
🔗
|
BlueMax |
ftp://ftp.fu-berlin.de/pc/games/idgames/levels/doom2/Ports/j-l/lstrealm.zip as an example. There's multiple copies of idgames out there. |
|
08:32
🔗
|
odie5533 |
If there's already multiple synchronized copies, why do you want to make another? |
|
08:32
🔗
|
BlueMax |
Why not? Never know what might happen. Good to have a central collection of everything we can find. |
|
08:33
🔗
|
odie5533 |
any idea how big the FTP is? |
|
08:34
🔗
|
BlueMax |
idgames is about 12GB I think |
|
08:35
🔗
|
BlueMax |
but that's NOT counting idstuff |
|
08:35
🔗
|
BlueMax |
which is a subdirectory I never downloaded |
|
08:35
🔗
|
BlueMax |
so it's probably a lot bigger. |
|
12:12
🔗
|
dashcloud |
Archive.org already has a collection of Doom stuff here: https://archive.org/details/doom-cds |
|
12:14
🔗
|
joepie91 |
is this being archived yet? http://modarchive.org/index.php |
|
12:15
🔗
|
BlueMax |
dashcloud, that isn't anywhere near all the Doom content that we know exists |
|
12:17
🔗
|
BlueMax |
I need sleep I'll be back to try and get this off the ground tomorrow sometime |
|
15:17
🔗
|
DFJustin |
yeah the doom cds collection is great but they stopped making them after about 1996: https://archive.org/browse.php?field=year&collection=doom-cds |
|
15:17
🔗
|
DFJustin |
whereas the community has kept on creating amazing stuff up to the present day |
|
15:33
🔗
|
SketchCow |
Doom-cds only solves one problem, yes. |
|
16:04
🔗
|
Schbirid |
i should copy all the quaddicted stuff to IA some day |
|
16:07
🔗
|
Schbirid |
and i wish properly archiving CDs was not such a pain in the ass |
|
16:14
🔗
|
joepie91 |
Schbirid: ? |
|
16:22
🔗
|
Schbirid |
joepie91: offsets, audiotracks, etc |
|
16:22
🔗
|
joepie91 |
ah |
|
16:22
🔗
|
Schbirid |
i ranted about that some months ago already :) |
|
16:22
🔗
|
joepie91 |
:) |
|
18:36
🔗
|
joepie91 |
http://www.computerweekly.com/blogs/public-sector/2013/11/conservatives-erase-internet-h.html |
|
18:51
🔗
|
M1das |
conservatives should be shot. |
|
18:56
🔗
|
* |
joepie91 raises eyebrow |
|
18:57
🔗
|
Schbirid |
that's taking a conservatite approach to the extreme ;) |
|
19:00
🔗
|
w0rp |
The modern MPs are fucks. |
|
19:00
🔗
|
w0rp |
They seem to be really trying hard to ruin the country, and now they also want to delete evidence of them doing that I guess. |
|
19:26
🔗
|
balrog |
joepie91: ...wow |
|
19:27
🔗
|
BiggieJon |
and I thought the US was fsck'd |
|
19:28
🔗
|
Schbirid |
you mean fucked |
|
19:29
🔗
|
Schbirid |
a fsck might be in order |
|
19:31
🔗
|
BiggieJon |
well, yes |
|
19:32
🔗
|
odie5533 |
"The bots grab web pages for the benefit of plebs" yay plebs! ... |
|
19:47
🔗
|
godane |
looks like you guys are going to be getting tons of images with my d-addicts.com dump |
|
19:50
🔗
|
godane |
a lot of the movie images are hosted on the site |
|
19:50
🔗
|
godane |
so there is going be less grabbing of other urls i guess |
|
20:07
🔗
|
yipdw |
well |
|
20:07
🔗
|
yipdw |
I wonder if IA's archivers crawl anyway and just dark robots.txt |
|
20:07
🔗
|
yipdw |
er, dark according to it |
|
20:07
🔗
|
yipdw |
or if they just never visit it at all |
|
20:07
🔗
|
yipdw |
I bet it's the latter |
|
20:07
🔗
|
yipdw |
underscor: ^^ |
|
20:07
🔗
|
yipdw |
too bad, we should have run that into archivebot |
|
20:11
🔗
|
ersi |
If it's been downloaded/crawled before, I'm sure they don't just remove it if they see a robots.txt |
|
20:11
🔗
|
ersi |
I'm sure it's just unavailable through Wayback Machine. |
|
20:11
🔗
|
ersi |
If the robot.txt is there at crawl time.. I have no idea. I kinda bet they won't keep crawling then |
|
20:14
🔗
|
balrog |
yipdw: they just never visit it at all -- it's bad behavior to crawl when the robots.txt is present |
|
20:14
🔗
|
balrog |
what IA should do imho is use the robots.txt AT THE TIME to determine whether the data will be visible |
|
20:14
🔗
|
balrog |
and require a written form to retroactively delete |
|
20:15
🔗
|
balrog |
domain squatters have been abusing retroactive robots.txt to black wide swaths out of IA |
|
20:15
🔗
|
ersi |
I'm sure that's why they do a robots.txt |
|
20:15
🔗
|
joepie91 |
godane: is d-addicts going down or something? |
|
20:16
🔗
|
ersi |
godane grabs everything he finds :) |
|
20:16
🔗
|
joepie91 |
:P |
|
20:16
🔗
|
joepie91 |
right. was just wondering if I missed a shutdown announcement |
|
20:16
🔗
|
ersi |
he's the human archivebot ^_^ |
|
20:16
🔗
|
joepie91 |
hehe |
|
20:16
🔗
|
joepie91 |
more like human heritrix |
|
20:18
🔗
|
yipdw |
a heritrix is a human |
|
20:18
🔗
|
yipdw |
a female, to be precise |
|
20:19
🔗
|
joepie91 |
? |
|
20:19
🔗
|
yipdw |
in English it refers to a female inheritor of something |
|
20:20
🔗
|
yipdw |
the name of the software is taken from that word |
|
20:20
🔗
|
yipdw |
:P |
|
20:20
🔗
|
yipdw |
just being pedantic |
|
20:20
🔗
|
ersi |
I didn't know that |
|
20:21
🔗
|
ersi |
I just read the same thing a couple of days ago, but about the Swedish equivalent of heritrix |
|
20:22
🔗
|
joepie91 |
oh lol |
|
20:30
🔗
|
w0rp |
I wish there was an ISP in the UK you could pay good money for to avoid traffic shaping. BT apparently doesn't like it when I upload 500GB in a couple of weeks. |
|
20:49
🔗
|
DFJustin |
I suppose it would be a heritor at that point |
|
21:08
🔗
|
godane |
so looks like Veronica Belmont left Tekzilla |
|
21:09
🔗
|
godane |
i'm uploading her last episode |
|
21:50
🔗
|
BlueMax |
is it just me or are there a lot less people oped lately |
|
21:52
🔗
|
balrog |
probably was a recent netsplit... |
|
21:52
🔗
|
balrog |
can someone up a bunch of us? |
|
21:52
🔗
|
balrog |
op* |
|
21:53
🔗
|
xmc |
have a snail hat |
|
21:56
🔗
|
BlueMax |
that's looking a lot better |
|
21:56
🔗
|
xmc |
mmm yeah |
|
21:56
🔗
|
BlueMax |
all dem green lights |
|
21:57
🔗
|
xmc |
green snail hats |
|
21:57
🔗
|
joepie91 |
:( |
|
21:58
🔗
|
* |
BlueMax pets joepie91 |
|
21:58
🔗
|
BlueMax |
some day. |
|
22:04
🔗
|
ersi |
lay of the green |
|
22:04
🔗
|
ersi |
I dun' see no' green |
|
22:07
🔗
|
BlueMax |
I be losin' my green |
|
22:08
🔗
|
BlueMax |
I'm envious of all you people who have persistent servers and shit |
|
22:11
🔗
|
touya |
heh |
|
22:11
🔗
|
touya |
you can get a VPS for as low as $5 i think |
|
22:12
🔗
|
BlueMax |
I'm an Australian with no job |
|
22:12
🔗
|
touya |
ask someone you know to give you an account for irc? |
|
22:12
🔗
|
touya |
not paying for this box either |
|
22:12
🔗
|
pft |
w/hois bluemax |
|
22:13
🔗
|
pft |
w/hois indeed |
|
22:13
🔗
|
BlueMax |
>___> |
|
22:14
🔗
|
BlueMax |
if I had the cash I'd get a dedicated server and start backing up Doom stuff |
|
22:27
🔗
|
godane |
so i got over 10k image files from d-addicts.com |
|
22:27
🔗
|
godane |
wayback magazine only has 1054 files |
|
22:28
🔗
|
godane |
this is based on images from d-addicts.com/forumfiles/ urls |
|
22:34
🔗
|
dashcloud |
BlueMax: if you've got a list, I'm happy to help out with it |
|
22:35
🔗
|
BlueMax |
alright gimme a sec |
|
22:41
🔗
|
BlueMax |
so anyway the first three targets for the Doom backup are the following 3 URLs |
|
22:41
🔗
|
BlueMax |
ftp://ftp.fu-berlin.de/pc/games/idgames |
|
22:41
🔗
|
BlueMax |
http://ftp.floorchan.org/wads/ |
|
22:41
🔗
|
BlueMax |
http://static.best-ever.org/wads/ |
|
22:42
🔗
|
BlueMax |
there's another source I want to get but I'm still working on that |
|
22:44
🔗
|
* |
BlueMax shaves SmileyG |
|
23:23
🔗
|
dashcloud |
I guess this is obvious to everyone else, but I've had much, much better computing experience once I dropped the swap partition entirely on my SSD- there aren't annoying hangs when memory runs short anymore |
|
23:25
🔗
|
joepie91 |
dashcloud: note that that might kill your SSD real quick |
|
23:26
🔗
|
dashcloud |
it may in fact have killed my previous SSD- it took 2 solid years though |
|
23:27
🔗
|
dashcloud |
luckily the warranty was for 3, so it wasn't that costly (except for some data when the SSD just crashed hard suddenly) |
|
23:29
🔗
|
balrog |
dashcloud: how much RAM do you have? |
|
23:29
🔗
|
balrog |
the fact that you need a swap partition to have reasonable performance indicates that you don't have enough RAM |
|
23:32
🔗
|
dashcloud |
I think I had it because I'd always had one before |
|
23:32
🔗
|
balrog |
so you got rid of it now? |
|
23:32
🔗
|
dashcloud |
yeah |
|
23:32
🔗
|
balrog |
ahh |
|
23:32
🔗
|
balrog |
I'm running OS X which doesn't let you disable swap... but it's currently using 95MB of 1GB of swap so I suppose I'm ok :p |
|
23:33
🔗
|
mistym |
10.9's pretty aggressive about never using swap, which is nice |
|
23:36
🔗
|
dashcloud |
is macports or homebrew the recommended choice now for installing tools and utilities? |
|
23:38
🔗
|
balrog |
it's really up to you; I used to use Macports but switched to Homebrew and like it a lot better |
|
23:38
🔗
|
balrog |
be warned, if you do ./configure && make && make install a lot, homebrew uses /usr/local by default |
|
23:38
🔗
|
pft |
i really perfer homebrew now |
|
23:38
🔗
|
balrog |
I recommend not doing make install with /usr/local as your prefix and using homebrew to manage /usr/local, but ... yeah |
|
23:39
🔗
|
mistym |
dashcloud: I'm biased :V |
|
23:40
🔗
|
dashcloud |
sorry- I'm not a current user of either, so I didn't know you were involved in one of the projects |
|
23:49
🔗
|
odie5533 |
How can you know if you need more RAM? |
|
23:50
🔗
|
balrog |
odie5533: if your inactive+free ram is frequently low |
|
23:50
🔗
|
balrog |
or if your "swap used" is high |
|
23:50
🔗
|
balrog |
(in activity monitor) |
|
23:56
🔗
|
odie5533 |
I dont think windows has a swap file. page file |
|
23:56
🔗
|
balrog |
oh you're on windows |
|
23:56
🔗
|
SketchCow |
Page file is a swap file. |
|
23:56
🔗
|
balrog |
yes |
|
23:57
🔗
|
odie5533 |
of course im on windows. only way to achieve maximum computing. |
|
23:57
🔗
|
balrog |
LOL |