| Time |
Nickname |
Message |
|
00:07
🔗
|
atphoenix |
Is the "archiveteam's choice" only one specific project? Or is it a mechanism that can allocate different active projects round-robin or based on various priority metrics to all the warriors? |
|
00:18
🔗
|
ScruffyB |
atphoenix, it is the currently active project. I don't think it supports round-robin. I think archive team relies on enough people running scripts manually to do more than one capture at once. |
|
00:20
🔗
|
ScruffyB |
markedL, I have only run the Virtualbox version: no working graph. The virutal box version runs docker anyway. |
|
00:32
🔗
|
atphoenix |
I was wondering about that because playstv and yahoogroups both were dying at about the same time |
|
00:33
🔗
|
atphoenix |
playstv was bandwidth heavy while yahoogroups needed time due to being slow and needing retries |
|
00:33
🔗
|
atphoenix |
I ran a couple warriors and manually assigned some to each, and 1 to urlteam |
|
00:33
🔗
|
atphoenix |
that way all the warriors stayed somewhat busy |
|
00:34
🔗
|
atphoenix |
ideally I'd suggest that the warriors be assigned tasks individually somewhat like that |
|
00:36
🔗
|
atphoenix |
group projects into low-bandwidth and high-bandwidth and by low-priority and high-priority |
|
00:37
🔗
|
atphoenix |
urlteam = low-bandwith/low-priority. playstv=high bw/high pri. Yahoogroups=low bw/high pri. Yahoo-groups-api(the files/photos effort)=high bw/high pri |
|
00:38
🔗
|
atphoenix |
and even better if a single warrior can be assigned units from different different projects |
|
01:07
🔗
|
|
bithippo has quit IRC (Textual IRC Client: www.textualapp.com) |
|
01:08
🔗
|
JAA |
Yeah, the network graph is broken only in the VM as far as I know. Something related to the VM's network adapter I think? |
|
01:10
🔗
|
JAA |
atphoenix: We would certainly welcome a PR that implements running multiple projects in the warrior. It's not an easy task though, and it gets tricky regarding system resources quickly. |
|
01:11
🔗
|
JAA |
But really the warrior is intended as a simple, low-barrier thing that anyone can run to help AT. |
|
01:11
🔗
|
JAA |
Power users will almost always want to run either the Docker containers or the plain scripts directly, in my experience. |
|
01:11
🔗
|
JAA |
The warrior adds too much overhead for running multiple in parallel. |
|
01:48
🔗
|
atphoenix |
hmm..I ran 4x warrior VMs each with 6 units on a 8 GB RAM Windows 10 box without issue. Core i5-3320m. |
|
01:49
🔗
|
atphoenix |
my limits were d/l and u/l bandwidth |
|
01:49
🔗
|
markedL |
they're slow to start because of the update mechanisms |
|
01:49
🔗
|
atphoenix |
I used the Warrior VM for simplicity on Windows |
|
01:50
🔗
|
atphoenix |
you mean the warrior update mechanisms? |
|
02:00
🔗
|
JAA |
Well ok, on Windows you might have to use the warrior, yeah. |
|
02:01
🔗
|
JAA |
And of course it will be fine if you have enough CPU power and RAM. It's just that it would be much more efficient under Linux with Docker or scripts since you avoid the virtualisation overhead for each VM. |
|
02:08
🔗
|
atphoenix |
I understand the efficiency side of the equation; I'm advocating for the ease-of-use side and that many recent systems have equal CPUs to what I listed and similar RAM. |
|
02:09
🔗
|
atphoenix |
I think we need to scale our abilities to grab sites and run archiving efforts |
|
02:10
🔗
|
astrid |
the original purpose of the warrior was to allow archiveteam to productively harness the fury of reddit |
|
02:11
🔗
|
|
DogsRNice has quit IRC (Read error: Connection reset by peer) |
|
02:13
🔗
|
atphoenix |
that sounds like what I'm talking about, but also scaling it further. Part of that also means being able to set monthly GB transfer limits and speed limits (kBps or Mbps) so that people can leave a Warrior running 24x7 without worrying about it eating all their bandwidth, whether measured in MB or Mbps. |
|
05:34
🔗
|
* |
ScruffyB was glad he now had "unlimited" bandwidth during the play.tv grab in the past I paid a ~$15 penalty for going over 60GB. |
|
05:51
🔗
|
|
tuluu has quit IRC (Remote host closed the connection) |
|
05:52
🔗
|
|
tuluu has joined #warrior |
|
05:57
🔗
|
ScruffyB |
lol, the warrior does not appear to have the drivers for the older, less efficinet network adapter. |
|
06:01
🔗
|
ScruffyB |
does have drivers for the more efficient "Paravirtualized network adapter (virtio-net)" |
|
06:26
🔗
|
|
Flashfire has quit IRC (Read error: Connection reset by peer) |
|
06:27
🔗
|
|
Flashfire has joined #warrior |
|
08:34
🔗
|
|
ShellyRol has quit IRC (Read error: Connection reset by peer) |
|
08:51
🔗
|
|
ShellyRol has joined #warrior |
|
09:54
🔗
|
|
zerkalo has quit IRC (Ping timeout: 258 seconds) |
|
09:54
🔗
|
|
mls has quit IRC (Ping timeout: 258 seconds) |
|
09:55
🔗
|
|
bilboed has quit IRC (Ping timeout: 258 seconds) |
|
09:55
🔗
|
|
luckcolor has quit IRC (Ping timeout: 258 seconds) |
|
09:55
🔗
|
|
luckcolor has joined #warrior |
|
09:59
🔗
|
|
mls has joined #warrior |
|
10:59
🔗
|
|
bilboed has joined #warrior |
|
13:46
🔗
|
|
chipmunk has quit IRC (Read error: Operation timed out) |
|
13:55
🔗
|
|
tuluu has quit IRC (Read error: Connection refused) |
|
13:55
🔗
|
|
tuluu has joined #warrior |
|
17:29
🔗
|
|
Ajay1 has quit IRC (Ping timeout: 276 seconds) |
|
17:39
🔗
|
|
Ajay1 has joined #warrior |
|
17:51
🔗
|
|
Ajay1 has quit IRC (Ping timeout: 276 seconds) |
|
18:15
🔗
|
|
Ajay1 has joined #warrior |
|
18:29
🔗
|
|
Ajay1 has quit IRC (Read error: Connection reset by peer) |
|
18:29
🔗
|
|
Ajay1 has joined #warrior |
|
19:55
🔗
|
|
mtntmnky has quit IRC (Remote host closed the connection) |
|
19:55
🔗
|
|
mtntmnky has joined #warrior |
|
20:43
🔗
|
|
DogsRNice has joined #warrior |
|
22:00
🔗
|
|
Ajay1 has quit IRC (Read error: Connection reset by peer) |
|
22:01
🔗
|
|
Ajay1 has joined #warrior |
|
22:20
🔗
|
|
Ajay1 has quit IRC (Quit: Ping timeout (120 seconds)) |
|
22:20
🔗
|
|
Ajay1 has joined #warrior |
|
23:01
🔗
|
|
Ajay1 has quit IRC (Quit: Ping timeout (120 seconds)) |
|
23:01
🔗
|
|
Ajay1 has joined #warrior |