#urlteam 2013-09-08,Sun

↑back Search

Time Nickname Message
05:20 🔗 xmc what the urlteam project needs is basically a large disk on a static ip
05:22 🔗 GLaDOS On a scale of 1 to 4T, with 1 being 1G, how large of a disk are we talking about?
05:25 🔗 xmc uh
05:25 🔗 xmc 1T or better
05:26 🔗 xmc latest torrent is 51G but that's rather heavily compressed
08:32 🔗 ersi Do we really need all of the old data, uncompressed?
10:58 🔗 soultcer Uncompressed old data?
11:01 🔗 ersi I'm not sure what xmc is hinting at, that's why I asked :)
11:04 🔗 ersi I mean, to just run the tracker - you don't need terabytes of storage. For seeding the releases, you'll need some storage however. If I recall correctly
11:12 🔗 soultcer Well you have the tracker, which is responsible for handing out tasks and temporarily storing the results. It needs a couple of GB storage, depending on how often you move the tasks off of it.
11:13 🔗 soultcer And then you have the actual database, which is about 700 GB or so, so better plan for at least 1 TB. The finished tasks from the tracker will be imported into the db. Then on release time you can create the release from the database
11:13 🔗 soultcer I always kept the finished tasks around until I made the next release, in case the db got lost/corrupted.
11:14 🔗 soultcer The DB can be restored/initialized from a release, and then you take all the finished tasks and import them.
11:20 🔗 ersi Ah, alright - wasn't familar with how the releases was generated - but that makes sense. If I've understood it correctly, one would at minimum have the database available when it's time for making releases. (So a very minimal setup could be just the tracker and temp store of results)
11:46 🔗 GLaDOS So, we'd need a box with at least 2TB (for futureproofing)?
11:50 🔗 GLaDOS soultcer: what was the CPU and RAM usage for it like?
11:55 🔗 GLaDOS Damn, 35EUR to install a 2TB external HDD, and 10EUR/month for it
12:26 🔗 soultcer GLaDOS: CPU is mostly needed for initial db import (due to db indexing) and then release generation (xz compression), but it is not necessary to have a extremly fast CPU
12:26 🔗 soultcer I used an AMD Athlon 64 X2 4800+ at 2x2,5 GHz, but most of the work is single-threaded.
12:27 🔗 GLaDOS I'm wondering whether it'd be better to get a 2T HDD added onto anarchive, or whether it'd be better just getting a whole new dedi
12:27 🔗 soultcer You don't necessarily need it on a dedi
12:27 🔗 GLaDOS http://fdcservers.net/dedicated_servers.php is pretty good, their HDD upgrade option is a one-time payment
12:27 🔗 GLaDOS Yeah, but I want something with a large enough HDD to hold it all
12:28 🔗 GLaDOS Guess what? No VPSes support that
12:28 🔗 soultcer The generated release (about 1/10th the size of database) can be uploaded to a seedbox. The database does not really need a fast or stable internet connection
12:29 🔗 BiggieJon my company is about to start offering dedicated hosting, too bad we wont be ready for a few more months
12:29 🔗 soultcer More important is to get a good fast HDD. I used a Velociraptor in the beginning and later switched to a Caviar Black when the database got bigger than the HDD. Both in RAID 1 configs
12:29 🔗 GLaDOS So approximately how much transfer is used for the DB?
12:29 🔗 soultcer You need to download the finished tasks, which is a couple of gigabytes per week as far as I remember. And then you need to upload the release once very 6 months, which is at the moment 75 gb
12:30 🔗 soultcer But of those 75 gb, most were the same as the 50 gb of the release before, so the upload was only a bit more than 25 gb.
12:30 🔗 soultcer In fact, the release creation process tries to leave as many files unchanged as possible, avoiding recompression of the same files and retransmission of the same data
12:30 🔗 GLaDOS I could manage that on my home VPS..
12:31 🔗 GLaDOS The DB, that is.
12:31 🔗 soultcer That's how I did it as well. Home server with some nice HDDs (but otherwise rather old hardware) worked just fine.
12:32 🔗 GLaDOS Difference between you and me is the average internet quality.
12:34 🔗 soultcer There is no need for a stable internet connection. You can download the tasks from the tracker using git-annex or rsync, and then do the upload of the release via rsync or bittorrent.
12:35 🔗 GLaDOS Alright
12:36 🔗 GLaDOS Now I just need to go search my cupboard for some decent sized HDDs.
12:37 🔗 GLaDOS Wait, I have a 1T mounted in my PC now.
13:09 🔗 GLaDOS SPEAKING OF LACK OF STABLE INTERNET CONNECTIONS
13:11 🔗 BiggieJon lol
21:24 🔗 xmc I could probably run it at my house
21:24 🔗 xmc I have quite stable neternets

irclogger-viewer