[00:48] *** ivan has quit IRC (Read error: Operation timed out) [00:49] *** kiska1 has quit IRC (Read error: Operation timed out) [00:49] *** ivan has joined #internetarchive [00:49] *** dxrt_ has quit IRC (Write error: Broken pipe) [00:50] *** Sunfused has quit IRC (Read error: Operation timed out) [00:54] *** iPlonk has quit IRC (Read error: Operation timed out) [00:54] *** iPlonk has joined #internetarchive [00:59] *** qw3rty111 has quit IRC (Read error: Operation timed out) [01:09] *** Sunfused has joined #internetarchive [01:10] *** Sunfused has quit IRC (Read error: Operation timed out) [01:20] *** kiska1 has joined #internetarchive [01:20] *** Sunfused has joined #internetarchive [01:21] *** qw3rty111 has joined #internetarchive [01:28] *** dxrt_ has joined #internetarchive [01:28] *** dxrt sets mode: +o dxrt_ [01:59] *** kiskabak has joined #internetarchive [04:15] *** odemg has quit IRC (Ping timeout: 615 seconds) [04:21] *** odemg has joined #internetarchive [04:26] *** bitBaron has quit IRC (Quit: My computer has gone to sleep. 😴😪ZZZzzz…) [04:43] *** qw3rty112 has joined #internetarchive [04:49] *** qw3rty111 has quit IRC (Read error: Operation timed out) [05:21] *** kiska1 has quit IRC (Ping timeout (120 seconds)) [05:22] *** kiska1 has joined #internetarchive [10:07] *** Phoen1x has joined #internetarchive [10:09] I'm in the process of uploading new dumps from an archive I help run to IA, and I'm only getting ~1-6 MB/s transfer speed. Is this roughly in line with expected performance? [10:16] Server is in Germany, so latency is probably playing a big factor [10:18] Yeah, another server in LA is getting 20 MB/s [10:19] Okay, any tips for how to increase throughput? I can paralellize uploads pretty easily, but what's a polite speed to aim for? [12:16] *** bitBaron has joined #internetarchive [12:25] Phoen1x: https://www.archiveteam.org/index.php?title=Internet_Archive#Upload_speed [12:31] Oh looks, there's helpful information right there on the main page. Maybe I should try this 'reading' stuff more often [12:31] Thanks Kaz [12:36] Ran the two commands in that section, and speed seems to have gone up a few MB/s [12:38] I think we've managed a fairly consistent 700-800Mbit/s out of EU before, but can't remember what the setup there way [12:38] was* [12:40] I'm seeing occasional bursts of up to ~150 Mb/s, but they don't last long [12:41] Running this from a small Hetzner vps [12:50] I rarely see over 100 Mb/s from my servers in Germany and France, FWIW. [12:51] And that's rsyncing to FOS. S3 uploads with "ia upload" are generally slower, less than 50 Mb/s typically. [15:07] *** bitBaron has quit IRC (My computer has gone to sleep. 😴😪ZZZzzz…) [15:38] *** bitBaron has joined #internetarchive [16:18] *** justas is now known as jut [16:39] *** bitBaron has quit IRC (Quit: My computer has gone to sleep. 😴😪ZZZzzz…) [17:48] Phoen1x: use BBR [17:48] ah it's even in the wiki that was linked [17:49] the important bits are net.core.default_qdisc=fq net.ipv4.tcp_congestion_control=bbr [18:26] *** picklefac has joined #internetarchive [18:26] *** picklefac has quit IRC (Connection closed) [18:29] *** picklefac has joined #internetarchive [18:53] *** yuitimoth has quit IRC (Read error: Operation timed out) [19:43] *** Jopik has quit IRC (Quit: Leaving) [19:47] *** Jopik has joined #internetarchive [19:47] *** Jopik has quit IRC (Remote host closed the connection) [19:47] *** Jopik has joined #internetarchive [21:20] *** yuitimoth has joined #internetarchive [21:29] ivan: please spell it out :) [21:29] I've never really managed to test that much from my EU machines [21:30] the results depend a lot on the network topology and status, in my experience, so it's hard to give one-size-fits-all solutions [21:30] spell what out? [21:30] ivan: which of the settings is most important (on the wiki) [21:31] I don't really want to test this or be responsible for it being right [21:32] I just don't fiddle with the numbers and get good throughput, but maybe that's just me :-) [21:42] what's on the wiki + high concurrency is what you really need to get decent speed [21:42] one upload just isn't going to max out the line [22:35] Yup, the only way I get sensible upload speeds from my ArchiveBot pipelines is by running multiple uploaders. Two are usually sufficient there to keep up with the inflow. [23:36] *** bitBaron has joined #internetarchive