| Time |
Nickname |
Message |
|
00:00
🔗
|
Kaz |
IA is currently chewing on 9.2 Gbit/s inbound and it's on fiiiiiiiiiiire |
|
00:35
🔗
|
|
Soni has joined #archiveteam-bs |
|
00:54
🔗
|
godane |
dashcloud: i got thur 2nd tape today |
|
00:58
🔗
|
godane |
i'm now capturing the best of benny hill |
|
01:04
🔗
|
|
n00b736 has joined #archiveteam-bs |
|
01:06
🔗
|
n00b736 |
Hello - I'm working on scraping sites that contain pre-smartphone mobile games (and both the sites and files are in danger of disappearing) and was wondering if anyone here can help either scripting or otherwise crowdsourcing this endeavor? |
|
01:07
🔗
|
n00b736 |
There are two specifically (a Chinese and Russian site) that appear to be in need of archiving, but they are vast... We are looking at 10,000+ games contained in .JAR and other filetypes. |
|
01:17
🔗
|
n00b736 |
Is this the right channel? |
|
01:18
🔗
|
|
sarahlynn has joined #archiveteam-bs |
|
01:20
🔗
|
|
sarahlynn has quit IRC (Read error: Connection reset by peer) |
|
01:21
🔗
|
Flashfire |
Bluemaxim |
|
01:21
🔗
|
Flashfire |
n00b736 Might be someone to speak to |
|
01:21
🔗
|
BlueMaxim |
hello gimme a minute |
|
01:22
🔗
|
n00b736 |
Jesus... Haha... Hey Blue - it's RetroRomper... Would it be better to message you on Discord? |
|
01:26
🔗
|
BlueMaxim |
yes |
|
01:27
🔗
|
|
m007a83 has quit IRC (Read error: Connection reset by peer) |
|
01:33
🔗
|
|
n00b736 has quit IRC (Quit: Page closed) |
|
01:36
🔗
|
|
m007a83 has joined #archiveteam-bs |
|
01:46
🔗
|
|
SimpBrain has quit IRC (Read error: Connection reset by peer) |
|
01:51
🔗
|
|
SimpBrain has joined #archiveteam-bs |
|
02:16
🔗
|
|
SimpBrain has quit IRC (Read error: Connection reset by peer) |
|
02:22
🔗
|
|
SimpBrain has joined #archiveteam-bs |
|
02:44
🔗
|
|
SimpBrain has quit IRC (Read error: Connection reset by peer) |
|
02:45
🔗
|
|
thejsa has quit IRC (Quit: No Ping reply in 180 seconds.) |
|
02:45
🔗
|
|
Soni has quit IRC (Quit: No Ping reply in 180 seconds.) |
|
02:48
🔗
|
|
thejsa has joined #archiveteam-bs |
|
02:50
🔗
|
|
SimpBrain has joined #archiveteam-bs |
|
02:51
🔗
|
|
SimpBrain has quit IRC (Remote host closed the connection) |
|
02:58
🔗
|
|
SimpBrain has joined #archiveteam-bs |
|
03:11
🔗
|
|
Dj-Wawa has quit IRC (Quit: Connection closed for inactivity) |
|
03:41
🔗
|
|
RichardG has quit IRC (Ping timeout: 252 seconds) |
|
03:43
🔗
|
|
wp494 has quit IRC (Ping timeout: 492 seconds) |
|
03:43
🔗
|
|
wp494 has joined #archiveteam-bs |
|
03:47
🔗
|
|
SimpBrain has quit IRC (Read error: Connection reset by peer) |
|
03:53
🔗
|
|
SimpBrain has joined #archiveteam-bs |
|
04:13
🔗
|
|
Stiletto has quit IRC (Ping timeout: 255 seconds) |
|
04:30
🔗
|
|
qw3rty115 has joined #archiveteam-bs |
|
04:35
🔗
|
|
qw3rty114 has quit IRC (Read error: Operation timed out) |
|
04:42
🔗
|
|
dhyan_nat has joined #archiveteam-bs |
|
04:46
🔗
|
|
odemgi has joined #archiveteam-bs |
|
04:48
🔗
|
|
odemgi_ has quit IRC (Ping timeout: 252 seconds) |
|
04:55
🔗
|
|
odemg has quit IRC (Ping timeout: 615 seconds) |
|
04:59
🔗
|
|
ndiddy has quit IRC () |
|
05:00
🔗
|
|
SimpBrain has quit IRC (Remote host closed the connection) |
|
05:01
🔗
|
|
odemg has joined #archiveteam-bs |
|
05:05
🔗
|
|
legoktm has joined #archiveteam-bs |
|
05:07
🔗
|
|
SimpBrain has joined #archiveteam-bs |
|
05:11
🔗
|
|
kbtoo_ has quit IRC (Read error: Connection reset by peer) |
|
05:14
🔗
|
|
kbtoo has joined #archiveteam-bs |
|
05:23
🔗
|
|
kbtoo has quit IRC (Read error: Connection reset by peer) |
|
05:26
🔗
|
|
kbtoo has joined #archiveteam-bs |
|
05:36
🔗
|
|
kbtoo has quit IRC (Read error: Connection reset by peer) |
|
05:39
🔗
|
|
kbtoo has joined #archiveteam-bs |
|
06:12
🔗
|
|
marked has quit IRC (Ping timeout: 255 seconds) |
|
06:13
🔗
|
|
marked has joined #archiveteam-bs |
|
06:20
🔗
|
|
Exairnous has quit IRC (Read error: Operation timed out) |
|
06:34
🔗
|
|
turnkit_ has quit IRC () |
|
06:43
🔗
|
|
MrRadar_ has joined #archiveteam-bs |
|
06:44
🔗
|
|
MrRadar has quit IRC (Read error: Operation timed out) |
|
06:54
🔗
|
|
MrRadar has joined #archiveteam-bs |
|
06:56
🔗
|
|
MrRadar_ has quit IRC (Read error: Operation timed out) |
|
07:25
🔗
|
|
SimpBrain has quit IRC (Remote host closed the connection) |
|
07:26
🔗
|
|
SimpBrain has joined #archiveteam-bs |
|
08:14
🔗
|
|
SimpBrain has quit IRC (Remote host closed the connection) |
|
08:15
🔗
|
|
SimpBrain has joined #archiveteam-bs |
|
08:40
🔗
|
|
killsushi has quit IRC (Quit: Leaving) |
|
08:55
🔗
|
|
RichardG has joined #archiveteam-bs |
|
09:20
🔗
|
|
SimpBrain has quit IRC (Read error: Connection reset by peer) |
|
09:20
🔗
|
|
SimpBrain has joined #archiveteam-bs |
|
09:24
🔗
|
|
Odd0002_ has joined #archiveteam-bs |
|
09:25
🔗
|
|
Odd0002 has quit IRC (Ping timeout: 252 seconds) |
|
09:25
🔗
|
|
Odd0002_ is now known as Odd0002 |
|
09:29
🔗
|
|
SimpBrain has quit IRC (Read error: Connection reset by peer) |
|
09:31
🔗
|
|
SimpBrain has joined #archiveteam-bs |
|
09:37
🔗
|
|
BlueMaxim has quit IRC (Quit: Leaving) |
|
09:54
🔗
|
|
JH88 has joined #archiveteam-bs |
|
10:09
🔗
|
Smiley |
I know I've covered this before but can we implement a bw limt on rsync? |
|
10:20
🔗
|
Smiley |
Surely it's just a step in the project pipeline, and we can pass a variable from the UI? |
|
10:56
🔗
|
|
atrocity has joined #archiveteam-bs |
|
11:23
🔗
|
|
dhyan_nat has quit IRC (Read error: Operation timed out) |
|
11:33
🔗
|
Smiley |
Item users:smap/01549/av: Step 4 of 8 Server returned 0 (HERR). Sleeping. |
|
11:33
🔗
|
Smiley |
Herr?! |
|
11:51
🔗
|
JAA |
"Header error", I believe. Some kind of error occurred while trying to read the response headers. |
|
11:51
🔗
|
JAA |
But I don't think these error codes are documented anywhere really. |
|
11:53
🔗
|
Smiley |
ah ok |
|
11:53
🔗
|
Smiley |
well reutrning 0 is derp |
|
11:53
🔗
|
Smiley |
Anyway, seems that B/W limits on the actual VM work now :) |
|
11:54
🔗
|
JAA |
Makes sense if you need to store the status code in an unsigned int since there is no status code 0 in HTTP. |
|
11:56
🔗
|
|
SmileyG has joined #archiveteam-bs |
|
12:02
🔗
|
|
bitBaron has joined #archiveteam-bs |
|
12:05
🔗
|
|
Smiley has quit IRC (Ping timeout: 615 seconds) |
|
12:27
🔗
|
|
VerifiedJ has quit IRC (Read error: Connection reset by peer) |
|
12:28
🔗
|
|
VerifiedJ has joined #archiveteam-bs |
|
12:40
🔗
|
|
wp494 has quit IRC (Ping timeout: 364 seconds) |
|
12:41
🔗
|
|
wp494 has joined #archiveteam-bs |
|
12:48
🔗
|
|
bitBaron has quit IRC (Quit: My computer has gone to sleep. 😴😪ZZZzzz…) |
|
13:23
🔗
|
|
MrRadar2 sets mode: +o MrRadar |
|
13:34
🔗
|
HCross |
SketchCow: is FoS still a thing? |
|
13:35
🔗
|
|
icedice has joined #archiveteam-bs |
|
13:48
🔗
|
|
MrRadar_ has joined #archiveteam-bs |
|
13:52
🔗
|
|
MrRadar has quit IRC (Read error: Operation timed out) |
|
13:55
🔗
|
|
benjinsmi has joined #archiveteam-bs |
|
13:57
🔗
|
|
benjins has quit IRC (Read error: Operation timed out) |
|
14:24
🔗
|
|
MrRadar_ is now known as MrRadar |
|
14:24
🔗
|
|
MrRadar2 sets mode: +o MrRadar |
|
14:33
🔗
|
|
Stiletto has joined #archiveteam-bs |
|
15:12
🔗
|
|
anarcat has left |
|
15:27
🔗
|
|
dhyan_nat has joined #archiveteam-bs |
|
15:35
🔗
|
|
bitBaron has joined #archiveteam-bs |
|
16:27
🔗
|
arkiver |
JAA: ok, was just wondeing |
|
16:27
🔗
|
arkiver |
wondering* |
|
17:06
🔗
|
|
schbirid has joined #archiveteam-bs |
|
17:38
🔗
|
|
Odd0002_ has joined #archiveteam-bs |
|
17:40
🔗
|
|
Odd0002 has quit IRC (Read error: Operation timed out) |
|
17:40
🔗
|
|
Odd0002_ is now known as Odd0002 |
|
17:44
🔗
|
|
icedice has quit IRC (Read error: Operation timed out) |
|
17:48
🔗
|
|
omarroth has joined #archiveteam-bs |
|
18:16
🔗
|
|
Exairnous has joined #archiveteam-bs |
|
18:29
🔗
|
|
SimpBrain has quit IRC (Read error: Connection reset by peer) |
|
18:29
🔗
|
|
SimpBrain has joined #archiveteam-bs |
|
18:51
🔗
|
|
Exairnous has quit IRC (Read error: Operation timed out) |
|
19:00
🔗
|
|
bitBaron has quit IRC (Quit: My computer has gone to sleep. 😴😪ZZZzzz…) |
|
19:03
🔗
|
|
omarroth has quit IRC (Read error: Connection reset by peer) |
|
19:06
🔗
|
|
omarroth has joined #archiveteam-bs |
|
19:11
🔗
|
|
icedice has joined #archiveteam-bs |
|
19:25
🔗
|
|
bitBaron has joined #archiveteam-bs |
|
19:26
🔗
|
|
SimpBrain has quit IRC (Read error: Connection reset by peer) |
|
19:26
🔗
|
|
S1mpbrain has joined #archiveteam-bs |
|
19:34
🔗
|
marked |
I'd prefer BW limits on rsync also. without qos, the other internet devices and downloader would be borked |
|
19:34
🔗
|
marked |
^know from experience |
|
19:52
🔗
|
godane |
dashcloud: so i may have another tape captured thats insync |
|
19:52
🔗
|
godane |
so that makes about 4 tapes and 3 of the tapes are 4+ hours |
|
20:28
🔗
|
|
BlueMax has joined #archiveteam-bs |
|
20:57
🔗
|
|
atrocity has quit IRC (Ping timeout: 246 seconds) |
|
21:17
🔗
|
godane |
dashcloud: i'm going to see about capturing one of the twilight zone tapes you made |
|
21:18
🔗
|
godane |
alot of the tapes sent was stuff i normally don't capture |
|
21:18
🔗
|
godane |
but i'm doing cause i just want to say the stuff was digitize |
|
21:23
🔗
|
|
dhyan_nat has quit IRC (Read error: Operation timed out) |
|
21:38
🔗
|
|
signius has joined #archiveteam-bs |
|
21:42
🔗
|
|
robbierut has joined #archiveteam-bs |
|
21:43
🔗
|
|
wp494 has quit IRC (Ping timeout: 364 seconds) |
|
21:44
🔗
|
|
wp494 has joined #archiveteam-bs |
|
21:47
🔗
|
marked |
Aside from Google minus, has there been another project with say more than 20 rsync targets? |
|
21:47
🔗
|
Kaz |
plenty |
|
21:48
🔗
|
|
Odd0002 has quit IRC (Ping timeout: 252 seconds) |
|
21:50
🔗
|
Kaz |
<marked> I'm not changing tracker code, that's why I called it an external load balancer instead |
|
21:50
🔗
|
robbierut |
Marked, you said not changing the tracker code, but you want to add another point of faillure in the system? Something before the tracker? |
|
21:50
🔗
|
Kaz |
how do you intend to do this without either modifying the tracker, or seesaw itself? |
|
21:51
🔗
|
|
Odd0002 has joined #archiveteam-bs |
|
21:52
🔗
|
marked |
Create a DNS record, Google minus.rsynctargets.archiveteam.org |
|
21:52
🔗
|
marked |
Match the same API as the current tracker |
|
21:53
🔗
|
marked |
Then in the Google minus pipeline script point to that DNS record |
|
21:53
🔗
|
marked |
The DNS record could point to the tracker or a load balancer |
|
21:54
🔗
|
|
astrid has joined #archiveteam-bs |
|
21:54
🔗
|
marked |
For projects that want the current tracker for rsync assignment things are the same |
|
21:55
🔗
|
marked |
For projects that have a target surge, the rsync assignment could be made from a different code base |
|
21:55
🔗
|
marked |
Seesaw and tracker code base does not need to be modified as rsync hearding needs change |
|
21:58
🔗
|
marked |
The matching methods mentioned by others is rsyncing to targets in the same data center, this should be easy to do |
|
21:59
🔗
|
marked |
There was more contraversy about matching by disk space or ASN, but the point is it's flexible enough to do whichever |
|
22:00
🔗
|
marked |
And the fallback is it can always go back to the tracker or duplicate what the tracker random assignment would do |
|
22:01
🔗
|
robbierut |
To be fair, I see what you want to do. But personally I think its better to have a target communicate with the tracker to automate the adding/removal of targets when they are full/empty. |
|
22:01
🔗
|
robbierut |
This would also lessen the babysitting from people |
|
22:02
🔗
|
robbierut |
No need for dns etc. Just 1 uplink from each target |
|
22:03
🔗
|
marked |
It's just a matter of time when the rsync assignments become a bottleneck again. Let the tracker do job assignment and confirmations. Those require the database and are critical |
|
22:03
🔗
|
marked |
The health stats is too complex to mix with tracker's other tasks |
|
22:05
🔗
|
marked |
Maybe someone knows for sure- does data center mapping improve total through put of the swarm? How would we do that with the tracker assignment of targets? |
|
22:05
🔗
|
|
atomicthu has quit IRC (Read error: Operation timed out) |
|
22:05
🔗
|
robbierut |
Marked: doesnt there need to be a better tracker then? I don't know but it doesnt seem as hard on the hardware as a target with all the incoming data, packing to big files and uploading again. I know the tracker had issues yesterday with a conmection limit, but is that artificial or is that actually the cpu/disk limit of the server also? |
|
22:05
🔗
|
|
omarroth has quit IRC (Read error: Connection reset by peer) |
|
22:06
🔗
|
marked |
I know it could be put in seesaw, but seesaw cannot be modified quickly |
|
22:08
🔗
|
marked |
I suppose I'm suggesting the two functions of job assignment and target assignment can be decoupled and should live in two code bases |
|
22:08
🔗
|
marked |
But since this is not needed all the time, only use it when there's a lot of targets |
|
22:09
🔗
|
marked |
Complexity when it's useful to get more throughout |
|
22:09
🔗
|
marked |
But turn it off when not needed |
|
22:09
🔗
|
|
atomicthu has joined #archiveteam-bs |
|
22:10
🔗
|
robbierut |
What would be the benefit of that compared to 1 very fast tracker? The ability to turn off is nice, but not really critical right? |
|
22:11
🔗
|
marked |
In one code base if the algorithm of assignment needs to change, doesn't the tracker need to restart too? |
|
22:14
🔗
|
robbierut |
Yeah but why would it need to change on a regular basis? |
|
22:19
🔗
|
marked |
I guess that part is partially due to the debate of what's a good assignment method. Sounds to me like it will be dynamic situation at least at first. |
|
22:26
🔗
|
marked |
I assume it would require try and see before the debate ended. If there was a single known good method beforehand that could be implemented initially. |
|
22:28
🔗
|
marked |
The next closest leaving the tracker the same and using bots to turn on and off targets. But this doesn't get the assignment preferences |
|
22:29
🔗
|
marked |
To do assignment preferences then you have to add extra health or topology data to the tracker and have the tracker utilize it or export it to seesaw or warrior for a decision there. |
|
22:30
🔗
|
marked |
Idk what that data is even exactly this moment but I don't want it in the tracker db or maybe don't want to send a full set to workers |
|
22:34
🔗
|
robbierut |
I'm not big in the code, but I sometimes see people saying "this target is taken off" or " this target is added". Don't know the exact mechanics but there probably is a list the tracker chooses from to send workers to. How hard would it be to let a target do that instead of a person? No need for constant health data, just 1 time to take off a target to let it empty and 1 time to put it on again. |
|
22:35
🔗
|
robbierut |
Ofcourse this won't fix the connection limit of active but very busy targets. But it would cut out quite some waiting to reassign a worker to a target thats not defenitly not accepting incoming. |
|
22:35
🔗
|
|
signius has quit IRC (Quit: Leaving) |
|
23:18
🔗
|
|
Despatche has joined #archiveteam-bs |
|
23:23
🔗
|
|
Exairnous has joined #archiveteam-bs |
|
23:51
🔗
|
|
ndiddy has joined #archiveteam-bs |