| Time |
Nickname |
Message |
|
05:20
🔗
|
xmc |
what the urlteam project needs is basically a large disk on a static ip |
|
05:22
🔗
|
GLaDOS |
On a scale of 1 to 4T, with 1 being 1G, how large of a disk are we talking about? |
|
05:25
🔗
|
xmc |
uh |
|
05:25
🔗
|
xmc |
1T or better |
|
05:26
🔗
|
xmc |
latest torrent is 51G but that's rather heavily compressed |
|
08:32
🔗
|
ersi |
Do we really need all of the old data, uncompressed? |
|
10:58
🔗
|
soultcer |
Uncompressed old data? |
|
11:01
🔗
|
ersi |
I'm not sure what xmc is hinting at, that's why I asked :) |
|
11:04
🔗
|
ersi |
I mean, to just run the tracker - you don't need terabytes of storage. For seeding the releases, you'll need some storage however. If I recall correctly |
|
11:12
🔗
|
soultcer |
Well you have the tracker, which is responsible for handing out tasks and temporarily storing the results. It needs a couple of GB storage, depending on how often you move the tasks off of it. |
|
11:13
🔗
|
soultcer |
And then you have the actual database, which is about 700 GB or so, so better plan for at least 1 TB. The finished tasks from the tracker will be imported into the db. Then on release time you can create the release from the database |
|
11:13
🔗
|
soultcer |
I always kept the finished tasks around until I made the next release, in case the db got lost/corrupted. |
|
11:14
🔗
|
soultcer |
The DB can be restored/initialized from a release, and then you take all the finished tasks and import them. |
|
11:20
🔗
|
ersi |
Ah, alright - wasn't familar with how the releases was generated - but that makes sense. If I've understood it correctly, one would at minimum have the database available when it's time for making releases. (So a very minimal setup could be just the tracker and temp store of results) |
|
11:46
🔗
|
GLaDOS |
So, we'd need a box with at least 2TB (for futureproofing)? |
|
11:50
🔗
|
GLaDOS |
soultcer: what was the CPU and RAM usage for it like? |
|
11:55
🔗
|
GLaDOS |
Damn, 35EUR to install a 2TB external HDD, and 10EUR/month for it |
|
12:26
🔗
|
soultcer |
GLaDOS: CPU is mostly needed for initial db import (due to db indexing) and then release generation (xz compression), but it is not necessary to have a extremly fast CPU |
|
12:26
🔗
|
soultcer |
I used an AMD Athlon 64 X2 4800+ at 2x2,5 GHz, but most of the work is single-threaded. |
|
12:27
🔗
|
GLaDOS |
I'm wondering whether it'd be better to get a 2T HDD added onto anarchive, or whether it'd be better just getting a whole new dedi |
|
12:27
🔗
|
soultcer |
You don't necessarily need it on a dedi |
|
12:27
🔗
|
GLaDOS |
http://fdcservers.net/dedicated_servers.php is pretty good, their HDD upgrade option is a one-time payment |
|
12:27
🔗
|
GLaDOS |
Yeah, but I want something with a large enough HDD to hold it all |
|
12:28
🔗
|
GLaDOS |
Guess what? No VPSes support that |
|
12:28
🔗
|
soultcer |
The generated release (about 1/10th the size of database) can be uploaded to a seedbox. The database does not really need a fast or stable internet connection |
|
12:29
🔗
|
BiggieJon |
my company is about to start offering dedicated hosting, too bad we wont be ready for a few more months |
|
12:29
🔗
|
soultcer |
More important is to get a good fast HDD. I used a Velociraptor in the beginning and later switched to a Caviar Black when the database got bigger than the HDD. Both in RAID 1 configs |
|
12:29
🔗
|
GLaDOS |
So approximately how much transfer is used for the DB? |
|
12:29
🔗
|
soultcer |
You need to download the finished tasks, which is a couple of gigabytes per week as far as I remember. And then you need to upload the release once very 6 months, which is at the moment 75 gb |
|
12:30
🔗
|
soultcer |
But of those 75 gb, most were the same as the 50 gb of the release before, so the upload was only a bit more than 25 gb. |
|
12:30
🔗
|
soultcer |
In fact, the release creation process tries to leave as many files unchanged as possible, avoiding recompression of the same files and retransmission of the same data |
|
12:30
🔗
|
GLaDOS |
I could manage that on my home VPS.. |
|
12:31
🔗
|
GLaDOS |
The DB, that is. |
|
12:31
🔗
|
soultcer |
That's how I did it as well. Home server with some nice HDDs (but otherwise rather old hardware) worked just fine. |
|
12:32
🔗
|
GLaDOS |
Difference between you and me is the average internet quality. |
|
12:34
🔗
|
soultcer |
There is no need for a stable internet connection. You can download the tasks from the tracker using git-annex or rsync, and then do the upload of the release via rsync or bittorrent. |
|
12:35
🔗
|
GLaDOS |
Alright |
|
12:36
🔗
|
GLaDOS |
Now I just need to go search my cupboard for some decent sized HDDs. |
|
12:37
🔗
|
GLaDOS |
Wait, I have a 1T mounted in my PC now. |
|
13:09
🔗
|
GLaDOS |
SPEAKING OF LACK OF STABLE INTERNET CONNECTIONS |
|
13:11
🔗
|
BiggieJon |
lol |
|
21:24
🔗
|
xmc |
I could probably run it at my house |
|
21:24
🔗
|
xmc |
I have quite stable neternets |