#internetarchive 2019-03-09,Sat

↑back Search

Time Nickname Message
00:48 🔗 ivan has quit IRC (Read error: Operation timed out)
00:49 🔗 kiska1 has quit IRC (Read error: Operation timed out)
00:49 🔗 ivan has joined #internetarchive
00:49 🔗 dxrt_ has quit IRC (Write error: Broken pipe)
00:50 🔗 Sunfused has quit IRC (Read error: Operation timed out)
00:54 🔗 iPlonk has quit IRC (Read error: Operation timed out)
00:54 🔗 iPlonk has joined #internetarchive
00:59 🔗 qw3rty111 has quit IRC (Read error: Operation timed out)
01:09 🔗 Sunfused has joined #internetarchive
01:10 🔗 Sunfused has quit IRC (Read error: Operation timed out)
01:20 🔗 kiska1 has joined #internetarchive
01:20 🔗 Sunfused has joined #internetarchive
01:21 🔗 qw3rty111 has joined #internetarchive
01:28 🔗 dxrt_ has joined #internetarchive
01:28 🔗 dxrt sets mode: +o dxrt_
01:59 🔗 kiskabak has joined #internetarchive
04:15 🔗 odemg has quit IRC (Ping timeout: 615 seconds)
04:21 🔗 odemg has joined #internetarchive
04:26 🔗 bitBaron has quit IRC (Quit: My computer has gone to sleep. 😴😪ZZZzzz…)
04:43 🔗 qw3rty112 has joined #internetarchive
04:49 🔗 qw3rty111 has quit IRC (Read error: Operation timed out)
05:21 🔗 kiska1 has quit IRC (Ping timeout (120 seconds))
05:22 🔗 kiska1 has joined #internetarchive
10:07 🔗 Phoen1x has joined #internetarchive
10:09 🔗 Phoen1x I'm in the process of uploading new dumps from an archive I help run to IA, and I'm only getting ~1-6 MB/s transfer speed. Is this roughly in line with expected performance?
10:16 🔗 Phoen1x Server is in Germany, so latency is probably playing a big factor
10:18 🔗 Phoen1x Yeah, another server in LA is getting 20 MB/s
10:19 🔗 Phoen1x Okay, any tips for how to increase throughput? I can paralellize uploads pretty easily, but what's a polite speed to aim for?
12:16 🔗 bitBaron has joined #internetarchive
12:25 🔗 Kaz Phoen1x: https://www.archiveteam.org/index.php?title=Internet_Archive#Upload_speed
12:31 🔗 Phoen1x Oh looks, there's helpful information right there on the main page. Maybe I should try this 'reading' stuff more often
12:31 🔗 Phoen1x Thanks Kaz
12:36 🔗 Phoen1x Ran the two commands in that section, and speed seems to have gone up a few MB/s
12:38 🔗 Kaz I think we've managed a fairly consistent 700-800Mbit/s out of EU before, but can't remember what the setup there way
12:38 🔗 Kaz was*
12:40 🔗 Phoen1x I'm seeing occasional bursts of up to ~150 Mb/s, but they don't last long
12:41 🔗 Phoen1x Running this from a small Hetzner vps
12:50 🔗 JAA I rarely see over 100 Mb/s from my servers in Germany and France, FWIW.
12:51 🔗 JAA And that's rsyncing to FOS. S3 uploads with "ia upload" are generally slower, less than 50 Mb/s typically.
15:07 🔗 bitBaron has quit IRC (My computer has gone to sleep. 😴😪ZZZzzz…)
15:38 🔗 bitBaron has joined #internetarchive
16:18 🔗 justas is now known as jut
16:39 🔗 bitBaron has quit IRC (Quit: My computer has gone to sleep. 😴😪ZZZzzz…)
17:48 🔗 ivan Phoen1x: use BBR
17:48 🔗 ivan ah it's even in the wiki that was linked
17:49 🔗 ivan the important bits are net.core.default_qdisc=fq net.ipv4.tcp_congestion_control=bbr
18:26 🔗 picklefac has joined #internetarchive
18:26 🔗 picklefac has quit IRC (Connection closed)
18:29 🔗 picklefac has joined #internetarchive
18:53 🔗 yuitimoth has quit IRC (Read error: Operation timed out)
19:43 🔗 Jopik has quit IRC (Quit: Leaving)
19:47 🔗 Jopik has joined #internetarchive
19:47 🔗 Jopik has quit IRC (Remote host closed the connection)
19:47 🔗 Jopik has joined #internetarchive
21:20 🔗 yuitimoth has joined #internetarchive
21:29 🔗 Nemo_bis ivan: please spell it out :)
21:29 🔗 Nemo_bis I've never really managed to test that much from my EU machines
21:30 🔗 Nemo_bis the results depend a lot on the network topology and status, in my experience, so it's hard to give one-size-fits-all solutions
21:30 🔗 ivan spell what out?
21:30 🔗 Nemo_bis ivan: which of the settings is most important (on the wiki)
21:31 🔗 ivan I don't really want to test this or be responsible for it being right
21:32 🔗 ivan I just don't fiddle with the numbers and get good throughput, but maybe that's just me :-)
21:42 🔗 Kaz what's on the wiki + high concurrency is what you really need to get decent speed
21:42 🔗 Kaz one upload just isn't going to max out the line
22:35 🔗 JAA Yup, the only way I get sensible upload speeds from my ArchiveBot pipelines is by running multiple uploaders. Two are usually sufficient there to keep up with the inflow.
23:36 🔗 bitBaron has joined #internetarchive

irclogger-viewer