[00:02] *** crwbot has quit IRC (Ping timeout: 255 seconds) [00:17] urghf... #ipfinite similarly is largely saturated :( [00:18] alembic: what do you mean by saturated? [00:19] *** odemg has quit IRC (Remote host closed the connection) [00:19] the rate-limiting + number of warriors pointed at it means my scripts get stuck at GetItemFromTracker until finally they kill themselves [00:20] I mean, it's a good thing for the project overall, but it also means my box is not being much help [00:20] *** odemg has joined #archiveteam-bs [00:21] alembic: how about URLTEAM? [00:21] yup, looks like the best target atm :) [00:21] Muad-Dib, got linked this video and a thank you note for our archival effort, thought you may want to see it as well: https://www.youtube.com/watch?v=o77BL-hCHxA [00:32] *** pizzaiolo has quit IRC (Ping timeout: 370 seconds) [00:33] *** pizzaiolo has joined #archiveteam-bs [01:02] live-updated trainwreck: https://twitter.com/gitlabstatus/status/826591961444384768 [01:02] "We accidentally deleted production data and might have to restore from backup." [01:10] *** crwbot_ has joined #archiveteam-bs [01:20] at least they *have* a backup [01:20] *** Frogging sets mode: +o joepie91 [01:30] rm -rf in production! Yay! [01:46] *** odemg has quit IRC (Remote host closed the connection) [01:47] *** odemg has joined #archiveteam-bs [01:51] *** pizzaiolo has quit IRC (Remote host closed the connection) [02:13] *** kristian_ has quit IRC (Remote host closed the connection) [02:19] are you guys considering scraping the app store? [02:19] https://arstechnica.com/apple/2017/01/future-ios-release-will-soon-end-support-for-unmaintained-32-bit-apps/ [02:19] Frogging: might have* [02:37] yeah, might have [02:38] :p [02:46] *** odemg has quit IRC (Remote host closed the connection) [02:47] *** odemg has joined #archiveteam-bs [02:57] *** odemg has quit IRC (Remote host closed the connection) [02:58] *** odemg has joined #archiveteam-bs [03:02] *** crwbot_ has quit IRC (Quit: leaving) [03:03] i'm going to see about doing a grab of wired.com [03:03] i found a way to grab all articles going back to 1993 [04:30] *** ndiddy has quit IRC (Read error: Connection reset by peer) [04:55] *** JensRex has quit IRC (Remote host closed the connection) [04:56] *** JensRex has joined #archiveteam-bs [05:03] *** VADemon has joined #archiveteam-bs [05:39] *** Sk1d has quit IRC (Ping timeout: 250 seconds) [05:45] *** Frogging has quit IRC (Read error: Operation timed out) [05:45] *** Frogging has joined #archiveteam-bs [05:46] *** yakfish has quit IRC (Read error: Operation timed out) [05:46] *** Coderjoe has quit IRC (Read error: Operation timed out) [05:46] *** chfoo has quit IRC (Read error: Operation timed out) [05:46] *** SadDM has quit IRC (Read error: Operation timed out) [05:46] *** Darkstar has quit IRC (Read error: Operation timed out) [05:46] *** jspiros has quit IRC (Read error: Operation timed out) [05:47] *** eprillios has quit IRC (Read error: Operation timed out) [05:47] *** rocode has quit IRC (Read error: Operation timed out) [05:47] *** Sk1d has joined #archiveteam-bs [05:47] *** yakfish has joined #archiveteam-bs [05:49] *** eprillios has joined #archiveteam-bs [05:50] *** kyounko has quit IRC (Read error: Operation timed out) [05:52] *** godane has quit IRC (Ping timeout: 492 seconds) [05:53] *** chfoo has joined #archiveteam-bs [05:56] *** Coderjoe has joined #archiveteam-bs [06:02] *** Darkstar has joined #archiveteam-bs [06:02] *** godane has joined #archiveteam-bs [06:29] *** VADemon has quit IRC (Read error: Connection reset by peer) [06:37] *** icedice has quit IRC (Quit: Leaving) [06:45] *** jspiros has joined #archiveteam-bs [06:45] *** rocode has joined #archiveteam-bs [06:47] *** SadDM has joined #archiveteam-bs [06:47] *** swebb sets mode: +o SadDM [07:02] *** odemg has quit IRC (Remote host closed the connection) [07:03] *** odemg has joined #archiveteam-bs [07:15] *** vitzli has joined #archiveteam-bs [07:21] *** Ravenloft has quit IRC (Read error: Connection reset by peer) [07:29] *** Stil3tt0 has joined #archiveteam-bs [07:30] *** Stil3tt0 has quit IRC (Client Quit) [07:42] *** User404 has joined #archiveteam-bs [07:45] *** odemg has quit IRC (Remote host closed the connection) [07:58] *** Aranje has quit IRC (Quit: Three sheets to the wind) [08:19] https://docs.google.com/document/d/1GCK53YDcBWQveod9kfzW-VCxIABGiryG7_z_6jHdVik/pub [08:19] TL;DR GitLab employee accidentally rm -rf'd the production database. [08:20] http://i.imgur.com/uCkqfgx.gif Backup your repos. [08:22] *** ItsYoda has quit IRC (Ping timeout: 260 seconds) [09:01] *** vitzli has quit IRC (Quit: Leaving) [09:17] repo copies is a major feature of git yes [09:20] it'd be cool to have some standard way to export issue and wiki data to the same repository, sort of like what fossil does [09:27] test your backups [09:29] Why wouldn't you have backups [10:03] *** GE has joined #archiveteam-bs [10:49] *** pizzaiolo has joined #archiveteam-bs [11:52] *** ravetcofx has quit IRC (Ping timeout: 506 seconds) [12:08] *** BlueMaxim has quit IRC (Quit: Leaving) [12:09] *** GE has quit IRC (Remote host closed the connection) [12:42] *** ItsYoda has joined #archiveteam-bs [13:10] Post on forum from 2007 with information I'm interested in: "Here is the solution to the problem ". 404 of course due to the 2014 purge. Fuck you ImageShack.Fuck you [13:19] *** SDr has quit IRC (hub.efnet.us ny.us.hub) [13:19] *** bsmith094 has quit IRC (hub.efnet.us ny.us.hub) [13:19] *** mundus201 has quit IRC (hub.efnet.us ny.us.hub) [13:19] *** midas1 has quit IRC (hub.efnet.us ny.us.hub) [13:19] *** achip has quit IRC (hub.efnet.us ny.us.hub) [13:19] *** SadDM has quit IRC (hub.efnet.us ny.us.hub) [13:19] *** rocode has quit IRC (hub.efnet.us ny.us.hub) [13:19] *** jspiros has quit IRC (hub.efnet.us ny.us.hub) [13:29] *** SadDM has joined #archiveteam-bs [13:29] *** rocode has joined #archiveteam-bs [13:29] *** jspiros has joined #archiveteam-bs [13:29] *** irc.colosolutions.net sets mode: +o SadDM [13:29] *** swebb sets mode: +o SadDM [14:18] zino_: :( [14:20] [Login to view this link] [14:51] *** RichardG_ has joined #archiveteam-bs [14:51] *** odemg has joined #archiveteam-bs [14:51] *** odemg has quit IRC (Connection closed) [14:52] *** RichardG has quit IRC (Ping timeout: 244 seconds) [15:01] *** odemg has joined #archiveteam-bs [15:13] https://www.theregister.co.uk/2017/02/01/gitlab_data_loss/ [15:14] *** VADemon has joined #archiveteam-bs [15:19] *** VADemon_ has joined #archiveteam-bs [15:22] Gitlab is currently livestreaming their data recovery: https://www.youtube.com/c/Gitlab/live [15:29] *** pizzaiolo has quit IRC (Ping timeout: 245 seconds) [15:31] *** VADemon has quit IRC (Read error: Operation timed out) [15:45] *** pizzaiolo has joined #archiveteam-bs [15:45] *** pizzaiolo has quit IRC (Remote host closed the connection) [15:46] *** pizzaiolo has joined #archiveteam-bs [16:00] TL;DR: XYZ decides to remove the directory. After a second or two he notices he ran it on , instead of [16:01] hey sysadmins are human [16:01] I'm pretty sure [16:02] Their backup methods didn't work and they didn't notice. until now. Luckily one of them made the last (6 hour old) backup manually [16:03] also servers were called db1... & db2... which maybe caused the mistake, imho [16:12] *** GE has joined #archiveteam-bs [16:12] *** SDr has joined #archiveteam-bs [16:12] *** bsmith094 has joined #archiveteam-bs [16:12] *** mundus201 has joined #archiveteam-bs [16:12] *** achip has joined #archiveteam-bs [16:12] *** midas1 has joined #archiveteam-bs [16:33] VADemon_: hmmmmm there's 2 production servers [16:34] db1 and db2 [16:34] he needed to clear the directory on db2 [16:34] he did it on db1 [16:34] Hell, at a old job while tired (undiagnosed sleep apnea) I managed to run a test script on a production system [16:34] The fact it let you do that is another issue. alas. [16:34] Yeah, and db2. was making problems because it lagged behind with replication because of evil -ruski-hackers- spammers [16:34] Importantly, he also did it in the middle of troubleshooting why db2 wasn't properly replicating from db1 [16:35] but then there was the 'oh restore from the 1 of 6 backup methods.... oh #1 doesn't work, try #2.... [16:35] and repeat upto #6 [16:35] They never actually checked the backups actually made anything [16:35] A great rule I read a year ago: TEST YOUR BACKUPS [16:36] they could at least have checked the backup size :D [16:36] Oh god ffs [16:36] so they got a list of thigns to do after fixing this isse [16:36] 4th... is 'setup alerting for backups' [16:36] 4th?! 4TH?!!! [16:36] yes. one of the files was "a few bytes" in size [16:37] VADemon_: I'm anal about these things even when it's got nothing to do with me [16:37] In my old job I'd randomly notice the wrong sized files etc and ask someone, even if it wasn't part of my role [16:40] *** VADemon_ is now known as VADemon [16:42] SmileyG: seeing this issue at (I think I can call it professional?) this level raises my self-confidense ;) [16:43] c* I've heard an alike story, one at Microsoft has mixed up different RDP terminals and ran some script on production [16:43] they ended up fixing it for the rest of the day [16:50] yup [16:50] im like 'wow either these guys really suck, or im at least semi professional [16:52] nintendo is killing dsiware http://forums.no-intro.org/viewtopic.php?f=8&t=2862 [16:59] *** schbirid has joined #archiveteam-bs [17:00] *** RichardG_ is now known as RichardG [17:28] *** fie_ has joined #archiveteam-bs [17:28] *** fie has quit IRC (Read error: Connection reset by peer) [17:42] SmileyG: I hope that your errors are so quickly publicized, so that we may armchair about them as well [17:43] gitlab folks are really nice, they let us have the domain and data for gitorious.org [17:43] so you shut your mouth SmileyG [17:45] i've been in situations where you're responsible for one large aspect (say, application performance) and then you're called in to address a production issue that looks like it has to do with your expertise. you fuck up in stress [17:45] this happens [17:45] obviously someone is going to learn something from this [17:45] a few years ago i probably would have made silly remarks about how this would never happen to me [17:45] i have never deleted anything in production [17:46] until uh it did [17:46] but i never have come anywhere near production either [17:46] i'm proud to say that i once had the opportunity to hit the Big Red Switch for microsoft maps, and i was able to keep my hands off it [17:46] it's super easy to sit back in retrospect and say "oh well I would have verified our restore strategy on a periodic basis", and if you do, congrats, that's a very nice place to be [17:47] in my experience it's difficult to argue time allocation to actually do that. until shit like this happens, and then it gets magically easier to schedule it in [17:47] probably just the way humans work I guess [17:48] we had a similar "oh, hey, now we have time for this" moment when heartbleed happened, and now suddenly that company has SSL certificate disaster recovery practices codified [17:51] my professional experience has been with small companies; experience at different scales will probably mean YMMV [17:51] *** crwbot has joined #archiveteam-bs [18:05] obligatory note [18:05] fuckups like the GitLab one happen *everywhere* [18:06] just most companies never own up to them and call them an "unexpected availability issue" while they frantically try to recover [18:06] (to put things into perspective) [18:07] also that [18:09] also, I recommend reading tech team blogs from big companies [18:09] you'll find that most of them have *no idea what they're doing* because they have to solve a new problem they haven't solved before [18:09] because the only viable way to learn the necessary skills in current-day software companies is to learn from your mistakes as you go... [18:11] tl;dr yes GitLab fucked up, but this is not abnormal compared to what happens every day and they're learning from their mistakes and publishing them, so I don't see the problem :P [18:11] ya [18:11] ^ [18:11] I applaud the transparency [18:15] definitely [19:11] *** kristian_ has joined #archiveteam-bs [19:14] *** odemg has quit IRC (Remote host closed the connection) [19:23] the nintendo dsi e-shop closes in march and there are still some unduped dsiware games, check http://forums.no-intro.org/viewtopic.php?f=8&t=2862 [19:38] *** odemg has joined #archiveteam-bs [19:38] Now might be a good time to think about archiving everything to do with the UK and the EU [19:38] Theresa May has just been given the go ahead to call Article 50 [19:38] FAK [19:38] caps, but still fak. [19:41] HCross: quick ELI5 on what is article 50? [19:41] the treaty that enables any EU state to leave the union [19:48] heh [19:48] *** odemg has quit IRC (Remote host closed the connection) [19:51] *** odemg has joined #archiveteam-bs [20:17] i love gitlab [20:31] *** kristian_ has quit IRC (Quit: Leaving) [20:51] schbirid: they fixed it yet? [20:52] no idea [21:04] yep, they're up as of a few hours ago SmileyG [21:28] *** BlueMaxim has joined #archiveteam-bs [21:28] HCross: really? [21:28] fuck common sense, I guess [21:32] joepie91, 498 votes to 114 [21:42] I mean it would be pretty undemocratic if they didn't [22:13] *** SDr has left [22:31] so i found another show from koreanet [22:31] here is a url for one of them: mms://vod2.kbs.gscdn.com/koreanet-2/gwangju/2008/tvpro/power/01/02.asf [22:31] nice [22:31] godane: have archivebot get it [22:32] can archivebot do mms streams [22:32] ? [22:33] cause i thought i couldn't [22:34] oh yeah I missed that part lol [22:42] *** schbirid has quit IRC (Quit: Leaving) [23:12] [22:42] I mean it would be pretty undemocratic if they didn't [23:12] uh, no, not at all [23:12] in fact, it's undemocratic that they *did* [23:12] despite clear signs of the referendum result being completely wrong [23:12] "democracy" isn't "blindly running the numbers": [23:12] to be clear, I'm not defending the brexit result [23:18] *** GE has quit IRC (Quit: zzz) [23:19] *** odemg has quit IRC (Remote host closed the connection) [23:23] *** odemg has joined #archiveteam-bs [23:58] the referendum itself wasn't wrong but they probably should have had a threshold higher than 50%