| Time |
Nickname |
Message |
|
00:48
🔗
|
|
ivan has quit IRC (Read error: Operation timed out) |
|
00:49
🔗
|
|
kiska1 has quit IRC (Read error: Operation timed out) |
|
00:49
🔗
|
|
ivan has joined #internetarchive |
|
00:49
🔗
|
|
dxrt_ has quit IRC (Write error: Broken pipe) |
|
00:50
🔗
|
|
Sunfused has quit IRC (Read error: Operation timed out) |
|
00:54
🔗
|
|
iPlonk has quit IRC (Read error: Operation timed out) |
|
00:54
🔗
|
|
iPlonk has joined #internetarchive |
|
00:59
🔗
|
|
qw3rty111 has quit IRC (Read error: Operation timed out) |
|
01:09
🔗
|
|
Sunfused has joined #internetarchive |
|
01:10
🔗
|
|
Sunfused has quit IRC (Read error: Operation timed out) |
|
01:20
🔗
|
|
kiska1 has joined #internetarchive |
|
01:20
🔗
|
|
Sunfused has joined #internetarchive |
|
01:21
🔗
|
|
qw3rty111 has joined #internetarchive |
|
01:28
🔗
|
|
dxrt_ has joined #internetarchive |
|
01:28
🔗
|
|
dxrt sets mode: +o dxrt_ |
|
01:59
🔗
|
|
kiskabak has joined #internetarchive |
|
04:15
🔗
|
|
odemg has quit IRC (Ping timeout: 615 seconds) |
|
04:21
🔗
|
|
odemg has joined #internetarchive |
|
04:26
🔗
|
|
bitBaron has quit IRC (Quit: My computer has gone to sleep. 😴😪ZZZzzz…) |
|
04:43
🔗
|
|
qw3rty112 has joined #internetarchive |
|
04:49
🔗
|
|
qw3rty111 has quit IRC (Read error: Operation timed out) |
|
05:21
🔗
|
|
kiska1 has quit IRC (Ping timeout (120 seconds)) |
|
05:22
🔗
|
|
kiska1 has joined #internetarchive |
|
10:07
🔗
|
|
Phoen1x has joined #internetarchive |
|
10:09
🔗
|
Phoen1x |
I'm in the process of uploading new dumps from an archive I help run to IA, and I'm only getting ~1-6 MB/s transfer speed. Is this roughly in line with expected performance? |
|
10:16
🔗
|
Phoen1x |
Server is in Germany, so latency is probably playing a big factor |
|
10:18
🔗
|
Phoen1x |
Yeah, another server in LA is getting 20 MB/s |
|
10:19
🔗
|
Phoen1x |
Okay, any tips for how to increase throughput? I can paralellize uploads pretty easily, but what's a polite speed to aim for? |
|
12:16
🔗
|
|
bitBaron has joined #internetarchive |
|
12:25
🔗
|
Kaz |
Phoen1x: https://www.archiveteam.org/index.php?title=Internet_Archive#Upload_speed |
|
12:31
🔗
|
Phoen1x |
Oh looks, there's helpful information right there on the main page. Maybe I should try this 'reading' stuff more often |
|
12:31
🔗
|
Phoen1x |
Thanks Kaz |
|
12:36
🔗
|
Phoen1x |
Ran the two commands in that section, and speed seems to have gone up a few MB/s |
|
12:38
🔗
|
Kaz |
I think we've managed a fairly consistent 700-800Mbit/s out of EU before, but can't remember what the setup there way |
|
12:38
🔗
|
Kaz |
was* |
|
12:40
🔗
|
Phoen1x |
I'm seeing occasional bursts of up to ~150 Mb/s, but they don't last long |
|
12:41
🔗
|
Phoen1x |
Running this from a small Hetzner vps |
|
12:50
🔗
|
JAA |
I rarely see over 100 Mb/s from my servers in Germany and France, FWIW. |
|
12:51
🔗
|
JAA |
And that's rsyncing to FOS. S3 uploads with "ia upload" are generally slower, less than 50 Mb/s typically. |
|
15:07
🔗
|
|
bitBaron has quit IRC (My computer has gone to sleep. 😴😪ZZZzzz…) |
|
15:38
🔗
|
|
bitBaron has joined #internetarchive |
|
16:18
🔗
|
|
justas is now known as jut |
|
16:39
🔗
|
|
bitBaron has quit IRC (Quit: My computer has gone to sleep. 😴😪ZZZzzz…) |
|
17:48
🔗
|
ivan |
Phoen1x: use BBR |
|
17:48
🔗
|
ivan |
ah it's even in the wiki that was linked |
|
17:49
🔗
|
ivan |
the important bits are net.core.default_qdisc=fq net.ipv4.tcp_congestion_control=bbr |
|
18:26
🔗
|
|
picklefac has joined #internetarchive |
|
18:26
🔗
|
|
picklefac has quit IRC (Connection closed) |
|
18:29
🔗
|
|
picklefac has joined #internetarchive |
|
18:53
🔗
|
|
yuitimoth has quit IRC (Read error: Operation timed out) |
|
19:43
🔗
|
|
Jopik has quit IRC (Quit: Leaving) |
|
19:47
🔗
|
|
Jopik has joined #internetarchive |
|
19:47
🔗
|
|
Jopik has quit IRC (Remote host closed the connection) |
|
19:47
🔗
|
|
Jopik has joined #internetarchive |
|
21:20
🔗
|
|
yuitimoth has joined #internetarchive |
|
21:29
🔗
|
Nemo_bis |
ivan: please spell it out :) |
|
21:29
🔗
|
Nemo_bis |
I've never really managed to test that much from my EU machines |
|
21:30
🔗
|
Nemo_bis |
the results depend a lot on the network topology and status, in my experience, so it's hard to give one-size-fits-all solutions |
|
21:30
🔗
|
ivan |
spell what out? |
|
21:30
🔗
|
Nemo_bis |
ivan: which of the settings is most important (on the wiki) |
|
21:31
🔗
|
ivan |
I don't really want to test this or be responsible for it being right |
|
21:32
🔗
|
ivan |
I just don't fiddle with the numbers and get good throughput, but maybe that's just me :-) |
|
21:42
🔗
|
Kaz |
what's on the wiki + high concurrency is what you really need to get decent speed |
|
21:42
🔗
|
Kaz |
one upload just isn't going to max out the line |
|
22:35
🔗
|
JAA |
Yup, the only way I get sensible upload speeds from my ArchiveBot pipelines is by running multiple uploaders. Two are usually sufficient there to keep up with the inflow. |
|
23:36
🔗
|
|
bitBaron has joined #internetarchive |