[00:03] *** Ctrl has joined #archiveteam [00:04] *** MMovie has quit IRC (Quit: Leaving.) [00:11] *** yawkat has quit IRC (Ping timeout: 604 seconds) [00:22] *** godane has quit IRC (Ping timeout: 506 seconds) [00:23] *** Jens has quit IRC (Remote host closed the connection) [00:24] *** Jens has joined #archiveteam [00:29] *** yawkat has joined #archiveteam [00:34] *** Mayonaise has quit IRC (Read error: Operation timed out) [00:39] *** godane has joined #archiveteam [00:56] *** Ctrl has quit IRC (Ping timeout: 268 seconds) [01:02] *** Ctrl has joined #archiveteam [01:20] *** MMovie has joined #archiveteam [01:53] *** Ctrl has quit IRC (Max SendQ exceeded) [01:56] *** Ctrl has joined #archiveteam [02:08] *** wp494 has quit IRC (Ping timeout: 252 seconds) [03:44] *** kode54 has quit IRC (Remote host closed the connection) [03:44] *** odemg has quit IRC (Ping timeout: 265 seconds) [03:44] *** kode54 has joined #archiveteam [03:48] *** qw3rty117 has joined #archiveteam [03:52] *** qw3rty116 has quit IRC (Ping timeout: 600 seconds) [03:56] *** odemg has joined #archiveteam [04:07] Might be good to try and figure out a list of what is hosted on legacy.gitbook.com and grab it: https://twitter.com/GitBookStatus/status/1141437901907144706 [04:08] It's the old version of a service, that went down for a while today. [04:08] It's not *that* high priority because all the stuff is also in git repos stored elsewhere, but it'd be nice to have in the Wayback Machine. [04:41] *** wyatt8740 has joined #archiveteam [05:22] *** kode54 has quit IRC (Quit: The Lounge - https://thelounge.chat) [05:24] *** kode54 has joined #archiveteam [05:34] *** spiff has joined #archiveteam [05:43] *** nataraj has joined #archiveteam [05:43] *** fuzzy8021 has quit IRC (Read error: Connection reset by peer) [05:45] *** fuzzy8021 has joined #archiveteam [06:15] *** tuluu has quit IRC (Ping timeout: 265 seconds) [06:17] *** tuluu has joined #archiveteam [06:22] *** deevious has joined #archiveteam [06:35] *** deevious has quit IRC (Quit: deevious) [06:52] *** deevious has joined #archiveteam [07:00] *** satoshi has quit IRC (Ping timeout: 265 seconds) [07:16] *** nataraj has quit IRC (Read error: Operation timed out) [07:26] *** nataraj has joined #archiveteam [07:57] *** stratum`_ has joined #archiveteam [07:57] *** cadbury has quit IRC (Ping timeout: 265 seconds) [07:57] *** spiff has quit IRC (Ping timeout: 265 seconds) [07:57] *** stratum` has quit IRC (Ping timeout: 265 seconds) [07:57] *** spifft has joined #archiveteam [07:57] *** spifft has quit IRC (Read error: Connection reset by peer) [07:57] *** Jon^ has quit IRC (Ping timeout: 265 seconds) [07:57] *** stratum`_ is now known as stratum` [07:57] *** jmtd has joined #archiveteam [07:58] *** burt_ has quit IRC (Ping timeout: 265 seconds) [07:58] *** spifft has joined #archiveteam [07:59] *** Laverne has quit IRC (Ping timeout: 265 seconds) [07:59] *** igloo251 has quit IRC (Ping timeout: 265 seconds) [08:00] *** omglolbah has quit IRC (Ping timeout: 265 seconds) [08:01] *** fuzzy8021 has quit IRC (Read error: Operation timed out) [08:01] *** mundus201 has quit IRC (Ping timeout: 265 seconds) [08:02] *** fuzzy8021 has joined #archiveteam [08:02] *** edsu has quit IRC (Ping timeout: 265 seconds) [08:02] *** edsu has joined #archiveteam [08:02] *** spifft has quit IRC (Ping timeout: 265 seconds) [08:02] *** lukeman has quit IRC (Ping timeout: 265 seconds) [08:02] *** wacky has quit IRC (Ping timeout: 265 seconds) [08:02] *** TC01 has quit IRC (Ping timeout: 265 seconds) [08:02] *** Gfy has quit IRC (Ping timeout: 265 seconds) [08:02] *** kiskabak has quit IRC (Ping timeout: 265 seconds) [08:03] *** omglolbah has joined #archiveteam [08:03] *** ats has quit IRC (Ping timeout: 265 seconds) [08:03] *** burt has joined #archiveteam [08:03] *** TC01 has joined #archiveteam [08:03] *** lukeman has joined #archiveteam [08:04] *** wacky has joined #archiveteam [08:04] *** ats has joined #archiveteam [08:09] *** mundus201 has joined #archiveteam [08:25] *** DLoader has joined #archiveteam [08:38] *** Mateon1 has quit IRC (Remote host closed the connection) [08:39] *** Mateon1 has joined #archiveteam [08:39] *** wacky has quit IRC (Read error: Operation timed out) [08:39] *** wacky_ has joined #archiveteam [08:39] *** Rondom has quit IRC (Read error: Operation timed out) [08:40] *** yawkat has quit IRC (Read error: Operation timed out) [08:41] *** nataraj has quit IRC (Read error: Operation timed out) [08:41] *** noirscape has quit IRC (Read error: Operation timed out) [08:41] *** brayden has quit IRC (Ping timeout: 268 seconds) [08:41] *** N4Y has quit IRC (Ping timeout: 268 seconds) [08:41] *** noirscape has joined #archiveteam [08:41] *** m007a83_ has joined #archiveteam [08:41] *** MrRadar2 has quit IRC (Read error: Operation timed out) [08:41] *** RichardG_ has joined #archiveteam [08:41] *** fennec has quit IRC (Read error: Operation timed out) [08:42] *** Ctrl has quit IRC (Ping timeout: 268 seconds) [08:42] *** Selanda_ has quit IRC (Ping timeout: 268 seconds) [08:42] *** randomdes has quit IRC (Ping timeout: 268 seconds) [08:42] *** bsmith093 has quit IRC (Ping timeout: 268 seconds) [08:42] *** fennec has joined #archiveteam [08:42] *** SmileyG has joined #archiveteam [08:42] *** oli has quit IRC (Read error: Operation timed out) [08:43] *** Hoolootwo has joined #archiveteam [08:43] *** af10b3e5e has joined #archiveteam [08:43] *** Selanda has joined #archiveteam [08:44] *** Xibalba has quit IRC (Ping timeout: 268 seconds) [08:44] *** odemgi_ has joined #archiveteam [08:44] *** nyaomi has quit IRC (Ping timeout: 268 seconds) [08:44] *** Rondom has joined #archiveteam [08:44] *** tsr has quit IRC (Ping timeout: 268 seconds) [08:44] *** BnAboyZ has quit IRC (Ping timeout: 268 seconds) [08:44] *** BnAboyZ has joined #archiveteam [08:44] *** Smiley has quit IRC (Ping timeout: 268 seconds) [08:44] *** voker57_ has quit IRC (Ping timeout: 365 seconds) [08:44] *** nataraj has joined #archiveteam [08:44] *** ats_ has joined #archiveteam [08:44] *** tsr has joined #archiveteam [08:45] *** Xibalba has joined #archiveteam [08:45] *** Fusl has quit IRC (Ping timeout: 365 seconds) [08:45] *** Wingy has quit IRC (Ping timeout: 268 seconds) [08:45] *** nyaomi has joined #archiveteam [08:45] *** ranma has quit IRC (Ping timeout: 252 seconds) [08:46] *** bsmith093 has joined #archiveteam [08:46] *** randomdes has joined #archiveteam [08:46] *** yawkat has joined #archiveteam [08:46] *** oli has joined #archiveteam [08:46] *** tuluu_ has joined #archiveteam [08:47] *** Coderjo_ has joined #archiveteam [08:47] *** noirscape has quit IRC (Ping timeout: 268 seconds) [08:47] *** Coderjo has quit IRC (Ping timeout: 365 seconds) [08:48] *** nataraj has quit IRC (Read error: Connection reset by peer) [08:48] *** colona_ has quit IRC (Ping timeout: 268 seconds) [08:48] *** noirscape has joined #archiveteam [08:49] *** nataraj has joined #archiveteam [08:49] *** Jeroen has quit IRC (Read error: Operation timed out) [08:49] *** BnAboyZ has quit IRC (Read error: Operation timed out) [08:49] *** brayden has joined #archiveteam [08:49] *** deevious has quit IRC (Ping timeout: 252 seconds) [08:49] *** d5f4a3622 has quit IRC (Ping timeout: 252 seconds) [08:49] *** Hooloovoo has quit IRC (Ping timeout: 252 seconds) [08:49] *** RichardG has quit IRC (Ping timeout: 252 seconds) [08:49] *** murmur has quit IRC (Ping timeout: 252 seconds) [08:49] *** pew has quit IRC (Ping timeout: 252 seconds) [08:49] *** ats has quit IRC (Ping timeout: 252 seconds) [08:49] *** burt has quit IRC (Ping timeout: 252 seconds) [08:49] *** tuluu has quit IRC (Ping timeout: 252 seconds) [08:49] *** odemgi has quit IRC (Ping timeout: 252 seconds) [08:49] *** seatsea has quit IRC (Ping timeout: 252 seconds) [08:49] *** z00nx has quit IRC (Ping timeout: 252 seconds) [08:49] *** kiska has quit IRC (Ping timeout: 252 seconds) [08:49] *** joshua_ has quit IRC (Ping timeout: 268 seconds) [08:49] *** Fusl has joined #archiveteam [08:49] *** BnAboyZ has joined #archiveteam [08:49] *** Fusl_ sets mode: +o Fusl [08:49] *** jut has quit IRC (Ping timeout: 252 seconds) [08:49] *** i0npulse has quit IRC (Ping timeout: 252 seconds) [08:49] *** m007a83 has quit IRC (Ping timeout: 252 seconds) [08:49] *** Flashfire has quit IRC (Ping timeout: 252 seconds) [08:49] *** coderobe has quit IRC (Ping timeout: 252 seconds) [08:49] *** Medo42 has quit IRC (Ping timeout: 252 seconds) [08:49] *** lukeman has quit IRC (Ping timeout: 252 seconds) [08:49] *** ColdIce has quit IRC (Ping timeout: 252 seconds) [08:49] *** Nemo_bis has quit IRC (Ping timeout: 252 seconds) [08:49] *** Wingy has joined #archiveteam [08:49] *** colona has joined #archiveteam [08:50] *** joshua_ has joined #archiveteam [08:50] *** Xibalba has quit IRC (Ping timeout: 268 seconds) [08:50] *** z00nx has joined #archiveteam [08:50] *** kiska has joined #archiveteam [08:50] *** lukeman has joined #archiveteam [08:50] *** Flashfire has joined #archiveteam [08:50] *** jut has joined #archiveteam [08:50] *** ColdIce has joined #archiveteam [08:50] *** Medo42 has joined #archiveteam [08:50] *** coderobe has joined #archiveteam [08:50] *** ranma has joined #archiveteam [08:50] *** svchfoo3 sets mode: +o kiska [08:51] *** svchfoo1 sets mode: +v Flashfire [08:51] *** seatsea has joined #archiveteam [08:51] *** deevious has joined #archiveteam [08:51] *** Nemo_bis has joined #archiveteam [08:51] *** MrRadar2 has joined #archiveteam [08:52] *** Xibalba has joined #archiveteam [08:52] *** N4Y has joined #archiveteam [08:52] *** i0npulse has joined #archiveteam [08:52] *** pew has joined #archiveteam [08:52] *** svchfoo3 sets mode: +o MrRadar2 [08:53] *** murmur has joined #archiveteam [08:55] *** Xibalba has quit IRC (Read error: Operation timed out) [08:55] *** voker57 has joined #archiveteam [08:57] *** Wingy has quit IRC (Ping timeout: 268 seconds) [08:57] *** Wingy has joined #archiveteam [08:58] *** Xibalba has joined #archiveteam [08:59] *** MrRadar2 has quit IRC (Remote host closed the connection) [08:59] *** MrRadar2 has joined #archiveteam [08:59] *** svchfoo1 sets mode: +o MrRadar2 [08:59] *** colona has quit IRC (Ping timeout: 268 seconds) [08:59] *** treora has quit IRC (Ping timeout: 268 seconds) [09:00] *** treora has joined #archiveteam [09:01] *** Fusl has quit IRC (Ping timeout: 365 seconds) [09:01] *** Ganonmast has quit IRC (Ping timeout: 268 seconds) [09:01] *** burt has joined #archiveteam [09:02] *** colona has joined #archiveteam [09:05] *** BnAboyZ has quit IRC (Ping timeout: 268 seconds) [09:05] *** BnAboyZ has joined #archiveteam [09:05] *** spifft has joined #archiveteam [09:06] *** Gfy has joined #archiveteam [09:06] *** Laverne has joined #archiveteam [09:06] *** igloo251 has joined #archiveteam [09:07] *** kiskabak has joined #archiveteam [09:07] *** cadbury has joined #archiveteam [09:08] *** svchfoo3 sets mode: +o kiskabak [09:08] *** yuitimoth has quit IRC (Ping timeout: 268 seconds) [09:08] *** yuitimoth has joined #archiveteam [09:09] *** Tenebrae has quit IRC (Ping timeout: 365 seconds) [09:10] *** nataraj has quit IRC (Read error: Connection reset by peer) [09:10] *** nataraj has joined #archiveteam [09:11] *** Tenebrae has joined #archiveteam [09:16] *** Fusl has joined #archiveteam [09:16] *** Fusl_ sets mode: +o Fusl [09:20] *** Ganonmast has joined #archiveteam [09:24] *** BlueMax has quit IRC (Quit: Leaving) [09:48] *** Ctrl has joined #archiveteam [09:58] *** VerifiedJ has joined #archiveteam [09:58] *** Stilettoo has quit IRC (Read error: Operation timed out) [09:59] *** Stiletto has joined #archiveteam [10:41] *** deevious has quit IRC (Quit: deevious) [10:51] *** cfarquhar has quit IRC (Read error: Operation timed out) [10:52] *** cfarquhar has joined #archiveteam [11:44] *** wp494 has joined #archiveteam [12:03] *** Hani has quit IRC (Quit: Hani) [12:24] *** Odd0002_ has joined #archiveteam [12:30] *** Odd0002 has quit IRC (Read error: Operation timed out) [12:30] *** Odd0002_ is now known as Odd0002 [12:47] *** deevious has joined #archiveteam [13:58] *** fredgido has quit IRC (Remote host closed the connection) [13:58] *** fredgido has joined #archiveteam [14:35] *** arbin has quit IRC (Ping timeout: 255 seconds) [14:35] *** arbin has joined #archiveteam [15:35] *** nataraj has quit IRC (Read error: Operation timed out) [15:35] *** spifft has quit IRC (Quit: spifft) [15:57] *** murmur has quit IRC (Quit: leaving) [15:58] *** murmur has joined #archiveteam [16:01] *** murmur has quit IRC (Client Quit) [16:27] *** murmur has joined #archiveteam [16:32] *** super3 has joined #archiveteam [16:32] hello [16:39] *** dan| has joined #archiveteam [16:46] *** dan- has quit IRC (Ping timeout: 615 seconds) [16:52] *** super3 has quit IRC (Ping timeout: 260 seconds) [16:53] *** martini has joined #archiveteam [16:59] *** benjinsmi has joined #archiveteam [17:04] *** benjins has quit IRC (Read error: Operation timed out) [17:30] *** susudo has joined #archiveteam [17:47] *** m007a83_ is now known as m007a83 [17:49] *** Mateon1 has quit IRC (Ping timeout: 360 seconds) [17:49] *** Mateon1 has joined #archiveteam [17:53] *** martini has quit IRC (Quit: No Reasson) [18:00] *** Jeroen has joined #archiveteam [18:00] *** spiff has joined #archiveteam [18:00] So, we can classify this as interesting? interesting. FOS Archivebot channel is now backed up to the tune of 4 terabytes [18:01] I checked, it's absolutely uploading [18:01] Not sure what's going on - maybe an outage happened back before, maybe something else. [18:01] But that's happening. I'm hoping it'll right itself. [18:05] S3 has been fairly shit recently, based on the moans I've seen in #newsgrabber [18:05] We're backed up to the tune of like 40TB ish iirc [18:05] 40TB? [18:05] i have like 100TB of newsgrabber stored [18:05] and yes, s3 is fairly shit recently [18:06] i'm getting a constant stream of 500 errors when uploading files [18:06] And we have like 800TB of googleminus that we have to get moving as well [18:18] I'll bring it up with operations [18:19] SketchCow: http://xor.meo.ws/0dd8abec/f3b9/4689/9333/97fe9808dcab.png no good and http://xor.meo.ws/e7ffd043/7cce/418c/985f/492734408995.png no good either [18:20] & absolutely no good: http://xor.meo.ws/d038a0fb/3139/4f42/993d/2626389c8dc4.png (shows how the throughput to the s3 interface is down to near zero since around two days) [18:21] so yeah, we're being backpressured by the s3 interface and therefore buffering a lot of data on archive team megawarc factory servers which we dont want [18:21] *** murmur has quit IRC (Quit: leaving) [18:22] *** murmur has joined #archiveteam [18:22] Brought up [18:22] thanks [18:23] "We had deployed a replacement kernel to mitigate the recent SACK linux vulnerabilities. Will roll that back (and instead rely on filtering for now to mitigate SACK)." [18:23] fwiw I think this is pre-SACK [18:23] but I guess every little helps [18:23] *** murmur has quit IRC (Client Quit) [18:24] exactly 46 hours is what i can tell from the graphs when everything turned upside down [18:25] *** murmur has joined #archiveteam [18:25] *** murmur has quit IRC (Client Quit) [18:26] *** murmur has joined #archiveteam [18:28] *** murmur has quit IRC (Client Quit) [18:31] *** murmur has joined #archiveteam [18:46] They said that exactly matches two kernel upgrades [18:46] They're reversing the easy one first [19:01] SketchCow: whatever they did, i guess the rollback, definitely increased the throughput by a lot again [19:01] we're back to similar levels as we were before, doing a 300-400 mbit/s to s3 now from abox-fks1 and 500ish mbit/s from abox-hel1 [19:27] I'll pass [19:28] along [19:55] Oh is s3 faster now? [19:55] its as slow as two days ago but its faster than the last 48 hours :P [19:55] I'll take that as an improvement :P [20:01] They're going to do more work on it [20:01] But this at least pulls it back [20:02] Thank you for checking up on it :-) [20:19] @textfiles - experimenting now with a much newer kernel on the s3 loadbalancer. again, will watch the stats but keep us posted... [20:20] SketchCow: can forward them this https://atdash.meo.ws/d/000000058/telegraf-detail?var-user=fusl&var-host=archivebox-hel1 and this https://atdash.meo.ws/d/000000058/telegraf-detail?var-user=fusl&var-host=archivebox-fks1 url [20:25] *** PhrackD has quit IRC (Read error: Operation timed out) [20:29] *** PhrackD has joined #archiveteam [20:36] *** murmur has quit IRC (Quit: leaving) [20:37] *** murmur has joined #archiveteam [21:36] *** squires has joined #archiveteam [21:56] *** susudo has quit IRC (Quit: Page closed) [22:52] *** MMovie has quit IRC (Quit: Leaving.) [22:58] *** balrog has quit IRC (Bye) [23:04] *** Ctrl has quit IRC (Ping timeout: 268 seconds) [23:24] *** VerifiedJ has quit IRC (Quit: Leaving) [23:40] *** ivan is now known as nothere [23:53] *** Ctrl has joined #archiveteam