| Time |
Nickname |
Message |
|
00:02
🔗
|
|
crwbot has quit IRC (Ping timeout: 255 seconds) |
|
00:17
🔗
|
alembic |
urghf... #ipfinite similarly is largely saturated :( |
|
00:18
🔗
|
pizzaiolo |
alembic: what do you mean by saturated? |
|
00:19
🔗
|
|
odemg has quit IRC (Remote host closed the connection) |
|
00:19
🔗
|
alembic |
the rate-limiting + number of warriors pointed at it means my scripts get stuck at GetItemFromTracker until finally they kill themselves |
|
00:20
🔗
|
alembic |
I mean, it's a good thing for the project overall, but it also means my box is not being much help |
|
00:20
🔗
|
|
odemg has joined #archiveteam-bs |
|
00:21
🔗
|
pizzaiolo |
alembic: how about URLTEAM? |
|
00:21
🔗
|
alembic |
yup, looks like the best target atm :) |
|
00:21
🔗
|
rocode |
Muad-Dib, got linked this video and a thank you note for our archival effort, thought you may want to see it as well: https://www.youtube.com/watch?v=o77BL-hCHxA |
|
00:32
🔗
|
|
pizzaiolo has quit IRC (Ping timeout: 370 seconds) |
|
00:33
🔗
|
|
pizzaiolo has joined #archiveteam-bs |
|
01:02
🔗
|
joepie91 |
live-updated trainwreck: https://twitter.com/gitlabstatus/status/826591961444384768 |
|
01:02
🔗
|
joepie91 |
"We accidentally deleted production data and might have to restore from backup." |
|
01:10
🔗
|
|
crwbot_ has joined #archiveteam-bs |
|
01:20
🔗
|
Frogging |
at least they *have* a backup |
|
01:20
🔗
|
|
Frogging sets mode: +o joepie91 |
|
01:30
🔗
|
JensRex |
rm -rf in production! Yay! |
|
01:46
🔗
|
|
odemg has quit IRC (Remote host closed the connection) |
|
01:47
🔗
|
|
odemg has joined #archiveteam-bs |
|
01:51
🔗
|
|
pizzaiolo has quit IRC (Remote host closed the connection) |
|
02:13
🔗
|
|
kristian_ has quit IRC (Remote host closed the connection) |
|
02:19
🔗
|
ndiddy |
are you guys considering scraping the app store? |
|
02:19
🔗
|
ndiddy |
https://arstechnica.com/apple/2017/01/future-ios-release-will-soon-end-support-for-unmaintained-32-bit-apps/ |
|
02:19
🔗
|
joepie91 |
Frogging: might have* |
|
02:37
🔗
|
Frogging |
yeah, might have |
|
02:38
🔗
|
Frogging |
:p |
|
02:46
🔗
|
|
odemg has quit IRC (Remote host closed the connection) |
|
02:47
🔗
|
|
odemg has joined #archiveteam-bs |
|
02:57
🔗
|
|
odemg has quit IRC (Remote host closed the connection) |
|
02:58
🔗
|
|
odemg has joined #archiveteam-bs |
|
03:02
🔗
|
|
crwbot_ has quit IRC (Quit: leaving) |
|
03:03
🔗
|
godane |
i'm going to see about doing a grab of wired.com |
|
03:03
🔗
|
godane |
i found a way to grab all articles going back to 1993 |
|
04:30
🔗
|
|
ndiddy has quit IRC (Read error: Connection reset by peer) |
|
04:55
🔗
|
|
JensRex has quit IRC (Remote host closed the connection) |
|
04:56
🔗
|
|
JensRex has joined #archiveteam-bs |
|
05:03
🔗
|
|
VADemon has joined #archiveteam-bs |
|
05:39
🔗
|
|
Sk1d has quit IRC (Ping timeout: 250 seconds) |
|
05:45
🔗
|
|
Frogging has quit IRC (Read error: Operation timed out) |
|
05:45
🔗
|
|
Frogging has joined #archiveteam-bs |
|
05:46
🔗
|
|
yakfish has quit IRC (Read error: Operation timed out) |
|
05:46
🔗
|
|
Coderjoe has quit IRC (Read error: Operation timed out) |
|
05:46
🔗
|
|
chfoo has quit IRC (Read error: Operation timed out) |
|
05:46
🔗
|
|
SadDM has quit IRC (Read error: Operation timed out) |
|
05:46
🔗
|
|
Darkstar has quit IRC (Read error: Operation timed out) |
|
05:46
🔗
|
|
jspiros has quit IRC (Read error: Operation timed out) |
|
05:47
🔗
|
|
eprillios has quit IRC (Read error: Operation timed out) |
|
05:47
🔗
|
|
rocode has quit IRC (Read error: Operation timed out) |
|
05:47
🔗
|
|
Sk1d has joined #archiveteam-bs |
|
05:47
🔗
|
|
yakfish has joined #archiveteam-bs |
|
05:49
🔗
|
|
eprillios has joined #archiveteam-bs |
|
05:50
🔗
|
|
kyounko has quit IRC (Read error: Operation timed out) |
|
05:52
🔗
|
|
godane has quit IRC (Ping timeout: 492 seconds) |
|
05:53
🔗
|
|
chfoo has joined #archiveteam-bs |
|
05:56
🔗
|
|
Coderjoe has joined #archiveteam-bs |
|
06:02
🔗
|
|
Darkstar has joined #archiveteam-bs |
|
06:02
🔗
|
|
godane has joined #archiveteam-bs |
|
06:29
🔗
|
|
VADemon has quit IRC (Read error: Connection reset by peer) |
|
06:37
🔗
|
|
icedice has quit IRC (Quit: Leaving) |
|
06:45
🔗
|
|
jspiros has joined #archiveteam-bs |
|
06:45
🔗
|
|
rocode has joined #archiveteam-bs |
|
06:47
🔗
|
|
SadDM has joined #archiveteam-bs |
|
06:47
🔗
|
|
swebb sets mode: +o SadDM |
|
07:02
🔗
|
|
odemg has quit IRC (Remote host closed the connection) |
|
07:03
🔗
|
|
odemg has joined #archiveteam-bs |
|
07:15
🔗
|
|
vitzli has joined #archiveteam-bs |
|
07:21
🔗
|
|
Ravenloft has quit IRC (Read error: Connection reset by peer) |
|
07:29
🔗
|
|
Stil3tt0 has joined #archiveteam-bs |
|
07:30
🔗
|
|
Stil3tt0 has quit IRC (Client Quit) |
|
07:42
🔗
|
|
User404 has joined #archiveteam-bs |
|
07:45
🔗
|
|
odemg has quit IRC (Remote host closed the connection) |
|
07:58
🔗
|
|
Aranje has quit IRC (Quit: Three sheets to the wind) |
|
08:19
🔗
|
rocode |
https://docs.google.com/document/d/1GCK53YDcBWQveod9kfzW-VCxIABGiryG7_z_6jHdVik/pub |
|
08:19
🔗
|
rocode |
TL;DR GitLab employee accidentally rm -rf'd the production database. |
|
08:20
🔗
|
rocode |
http://i.imgur.com/uCkqfgx.gif Backup your repos. |
|
08:22
🔗
|
|
ItsYoda has quit IRC (Ping timeout: 260 seconds) |
|
09:01
🔗
|
|
vitzli has quit IRC (Quit: Leaving) |
|
09:17
🔗
|
yipdw |
repo copies is a major feature of git yes |
|
09:20
🔗
|
yipdw |
it'd be cool to have some standard way to export issue and wiki data to the same repository, sort of like what fossil does |
|
09:27
🔗
|
RedType |
test your backups |
|
09:29
🔗
|
eientei95 |
Why wouldn't you have backups |
|
10:03
🔗
|
|
GE has joined #archiveteam-bs |
|
10:49
🔗
|
|
pizzaiolo has joined #archiveteam-bs |
|
11:52
🔗
|
|
ravetcofx has quit IRC (Ping timeout: 506 seconds) |
|
12:08
🔗
|
|
BlueMaxim has quit IRC (Quit: Leaving) |
|
12:09
🔗
|
|
GE has quit IRC (Remote host closed the connection) |
|
12:42
🔗
|
|
ItsYoda has joined #archiveteam-bs |
|
13:10
🔗
|
zino_ |
Post on forum from 2007 with information I'm interested in: "Here is the solution to the problem <link to jpg om imageshack.us>". 404 of course due to the 2014 purge. Fuck you ImageShack.Fuck you |
|
13:19
🔗
|
|
SDr has quit IRC (hub.efnet.us ny.us.hub) |
|
13:19
🔗
|
|
bsmith094 has quit IRC (hub.efnet.us ny.us.hub) |
|
13:19
🔗
|
|
mundus201 has quit IRC (hub.efnet.us ny.us.hub) |
|
13:19
🔗
|
|
midas1 has quit IRC (hub.efnet.us ny.us.hub) |
|
13:19
🔗
|
|
achip has quit IRC (hub.efnet.us ny.us.hub) |
|
13:19
🔗
|
|
SadDM has quit IRC (hub.efnet.us ny.us.hub) |
|
13:19
🔗
|
|
rocode has quit IRC (hub.efnet.us ny.us.hub) |
|
13:19
🔗
|
|
jspiros has quit IRC (hub.efnet.us ny.us.hub) |
|
13:29
🔗
|
|
SadDM has joined #archiveteam-bs |
|
13:29
🔗
|
|
rocode has joined #archiveteam-bs |
|
13:29
🔗
|
|
jspiros has joined #archiveteam-bs |
|
13:29
🔗
|
|
irc.colosolutions.net sets mode: +o SadDM |
|
13:29
🔗
|
|
swebb sets mode: +o SadDM |
|
14:18
🔗
|
pizzaiolo |
zino_: :( |
|
14:20
🔗
|
Aoede |
[Login to view this link] |
|
14:51
🔗
|
|
RichardG_ has joined #archiveteam-bs |
|
14:51
🔗
|
|
odemg has joined #archiveteam-bs |
|
14:51
🔗
|
|
odemg has quit IRC (Connection closed) |
|
14:52
🔗
|
|
RichardG has quit IRC (Ping timeout: 244 seconds) |
|
15:01
🔗
|
|
odemg has joined #archiveteam-bs |
|
15:13
🔗
|
Aoede |
https://www.theregister.co.uk/2017/02/01/gitlab_data_loss/ |
|
15:14
🔗
|
|
VADemon has joined #archiveteam-bs |
|
15:19
🔗
|
|
VADemon_ has joined #archiveteam-bs |
|
15:22
🔗
|
MrRadar |
Gitlab is currently livestreaming their data recovery: https://www.youtube.com/c/Gitlab/live |
|
15:29
🔗
|
|
pizzaiolo has quit IRC (Ping timeout: 245 seconds) |
|
15:31
🔗
|
|
VADemon has quit IRC (Read error: Operation timed out) |
|
15:45
🔗
|
|
pizzaiolo has joined #archiveteam-bs |
|
15:45
🔗
|
|
pizzaiolo has quit IRC (Remote host closed the connection) |
|
15:46
🔗
|
|
pizzaiolo has joined #archiveteam-bs |
|
16:00
🔗
|
VADemon_ |
TL;DR: XYZ decides to remove the directory. After a second or two he notices he ran it on <PRODUCATION>, instead of <PROBLEMATIC DB> |
|
16:01
🔗
|
pizzaiolo |
hey sysadmins are human |
|
16:01
🔗
|
pizzaiolo |
I'm pretty sure |
|
16:02
🔗
|
VADemon_ |
Their backup methods didn't work and they didn't notice. until now. Luckily one of them made the last (6 hour old) backup manually |
|
16:03
🔗
|
VADemon_ |
also servers were called db1... & db2... which maybe caused the mistake, imho |
|
16:12
🔗
|
|
GE has joined #archiveteam-bs |
|
16:12
🔗
|
|
SDr has joined #archiveteam-bs |
|
16:12
🔗
|
|
bsmith094 has joined #archiveteam-bs |
|
16:12
🔗
|
|
mundus201 has joined #archiveteam-bs |
|
16:12
🔗
|
|
achip has joined #archiveteam-bs |
|
16:12
🔗
|
|
midas1 has joined #archiveteam-bs |
|
16:33
🔗
|
SmileyG |
VADemon_: hmmmmm there's 2 production servers |
|
16:34
🔗
|
SmileyG |
db1 and db2 |
|
16:34
🔗
|
SmileyG |
he needed to clear the directory on db2 |
|
16:34
🔗
|
SmileyG |
he did it on db1 |
|
16:34
🔗
|
SmileyG |
Hell, at a old job while tired (undiagnosed sleep apnea) I managed to run a test script on a production system |
|
16:34
🔗
|
SmileyG |
The fact it let you do that is another issue. alas. |
|
16:34
🔗
|
VADemon_ |
Yeah, and db2. was making problems because it lagged behind with replication because of evil -ruski-hackers- spammers |
|
16:34
🔗
|
MrRadar |
Importantly, he also did it in the middle of troubleshooting why db2 wasn't properly replicating from db1 |
|
16:35
🔗
|
SmileyG |
but then there was the 'oh restore from the 1 of 6 backup methods.... oh #1 doesn't work, try #2.... |
|
16:35
🔗
|
SmileyG |
and repeat upto #6 |
|
16:35
🔗
|
SmileyG |
They never actually checked the backups actually made anything |
|
16:35
🔗
|
VADemon_ |
A great rule I read a year ago: TEST YOUR BACKUPS |
|
16:36
🔗
|
VADemon_ |
they could at least have checked the backup size :D |
|
16:36
🔗
|
SmileyG |
Oh god ffs |
|
16:36
🔗
|
SmileyG |
so they got a list of thigns to do after fixing this isse |
|
16:36
🔗
|
SmileyG |
4th... is 'setup alerting for backups' |
|
16:36
🔗
|
SmileyG |
4th?! 4TH?!!! |
|
16:36
🔗
|
VADemon_ |
yes. one of the files was "a few bytes" in size |
|
16:37
🔗
|
SmileyG |
VADemon_: I'm anal about these things even when it's got nothing to do with me |
|
16:37
🔗
|
SmileyG |
In my old job I'd randomly notice the wrong sized files etc and ask someone, even if it wasn't part of my role |
|
16:40
🔗
|
|
VADemon_ is now known as VADemon |
|
16:42
🔗
|
VADemon |
SmileyG: seeing this issue at (I think I can call it professional?) this level raises my self-confidense ;) |
|
16:43
🔗
|
VADemon |
c* I've heard an alike story, one at Microsoft has mixed up different RDP terminals and ran some script on production |
|
16:43
🔗
|
VADemon |
they ended up fixing it for the rest of the day |
|
16:50
🔗
|
SmileyG |
yup |
|
16:50
🔗
|
SmileyG |
im like 'wow either these guys really suck, or im at least semi professional |
|
16:52
🔗
|
balrog |
nintendo is killing dsiware http://forums.no-intro.org/viewtopic.php?f=8&t=2862 |
|
16:59
🔗
|
|
schbirid has joined #archiveteam-bs |
|
17:00
🔗
|
|
RichardG_ is now known as RichardG |
|
17:28
🔗
|
|
fie_ has joined #archiveteam-bs |
|
17:28
🔗
|
|
fie has quit IRC (Read error: Connection reset by peer) |
|
17:42
🔗
|
yipdw |
SmileyG: I hope that your errors are so quickly publicized, so that we may armchair about them as well |
|
17:43
🔗
|
xmc |
gitlab folks are really nice, they let us have the domain and data for gitorious.org |
|
17:43
🔗
|
xmc |
so you shut your mouth SmileyG |
|
17:45
🔗
|
yipdw |
i've been in situations where you're responsible for one large aspect (say, application performance) and then you're called in to address a production issue that looks like it has to do with your expertise. you fuck up in stress |
|
17:45
🔗
|
yipdw |
this happens |
|
17:45
🔗
|
yipdw |
obviously someone is going to learn something from this |
|
17:45
🔗
|
yipdw |
a few years ago i probably would have made silly remarks about how this would never happen to me |
|
17:45
🔗
|
xmc |
i have never deleted anything in production |
|
17:46
🔗
|
yipdw |
until uh it did |
|
17:46
🔗
|
xmc |
but i never have come anywhere near production either |
|
17:46
🔗
|
xmc |
i'm proud to say that i once had the opportunity to hit the Big Red Switch for microsoft maps, and i was able to keep my hands off it |
|
17:46
🔗
|
yipdw |
it's super easy to sit back in retrospect and say "oh well I would have verified our restore strategy on a periodic basis", and if you do, congrats, that's a very nice place to be |
|
17:47
🔗
|
yipdw |
in my experience it's difficult to argue time allocation to actually do that. until shit like this happens, and then it gets magically easier to schedule it in |
|
17:47
🔗
|
yipdw |
probably just the way humans work I guess |
|
17:48
🔗
|
yipdw |
we had a similar "oh, hey, now we have time for this" moment when heartbleed happened, and now suddenly that company has SSL certificate disaster recovery practices codified |
|
17:51
🔗
|
yipdw |
my professional experience has been with small companies; experience at different scales will probably mean YMMV |
|
17:51
🔗
|
|
crwbot has joined #archiveteam-bs |
|
18:05
🔗
|
joepie91 |
obligatory note |
|
18:05
🔗
|
joepie91 |
fuckups like the GitLab one happen *everywhere* |
|
18:06
🔗
|
joepie91 |
just most companies never own up to them and call them an "unexpected availability issue" while they frantically try to recover |
|
18:06
🔗
|
joepie91 |
(to put things into perspective) |
|
18:07
🔗
|
xmc |
also that |
|
18:09
🔗
|
joepie91 |
also, I recommend reading tech team blogs from big companies |
|
18:09
🔗
|
joepie91 |
you'll find that most of them have *no idea what they're doing* because they have to solve a new problem they haven't solved before |
|
18:09
🔗
|
joepie91 |
because the only viable way to learn the necessary skills in current-day software companies is to learn from your mistakes as you go... |
|
18:11
🔗
|
joepie91 |
tl;dr yes GitLab fucked up, but this is not abnormal compared to what happens every day and they're learning from their mistakes and publishing them, so I don't see the problem :P |
|
18:11
🔗
|
xmc |
ya |
|
18:11
🔗
|
yipdw |
^ |
|
18:11
🔗
|
Frogging |
I applaud the transparency |
|
18:15
🔗
|
pizzaiolo |
definitely |
|
19:11
🔗
|
|
kristian_ has joined #archiveteam-bs |
|
19:14
🔗
|
|
odemg has quit IRC (Remote host closed the connection) |
|
19:23
🔗
|
Sanqui |
the nintendo dsi e-shop closes in march and there are still some unduped dsiware games, check http://forums.no-intro.org/viewtopic.php?f=8&t=2862 |
|
19:38
🔗
|
|
odemg has joined #archiveteam-bs |
|
19:38
🔗
|
HCross |
Now might be a good time to think about archiving everything to do with the UK and the EU |
|
19:38
🔗
|
HCross |
Theresa May has just been given the go ahead to call Article 50 |
|
19:38
🔗
|
Kaz |
FAK |
|
19:38
🔗
|
Kaz |
caps, but still fak. |
|
19:41
🔗
|
pizzaiolo |
HCross: quick ELI5 on what is article 50? |
|
19:41
🔗
|
HCross |
the treaty that enables any EU state to leave the union |
|
19:48
🔗
|
Frogging |
heh |
|
19:48
🔗
|
|
odemg has quit IRC (Remote host closed the connection) |
|
19:51
🔗
|
|
odemg has joined #archiveteam-bs |
|
20:17
🔗
|
schbirid |
i love gitlab |
|
20:31
🔗
|
|
kristian_ has quit IRC (Quit: Leaving) |
|
20:51
🔗
|
SmileyG |
schbirid: they fixed it yet? |
|
20:52
🔗
|
schbirid |
no idea |
|
21:04
🔗
|
Kaz |
yep, they're up as of a few hours ago SmileyG |
|
21:28
🔗
|
|
BlueMaxim has joined #archiveteam-bs |
|
21:28
🔗
|
joepie91 |
HCross: really? |
|
21:28
🔗
|
joepie91 |
fuck common sense, I guess |
|
21:32
🔗
|
HCross |
joepie91, 498 votes to 114 |
|
21:42
🔗
|
pizzaiolo |
I mean it would be pretty undemocratic if they didn't |
|
22:13
🔗
|
|
SDr has left |
|
22:31
🔗
|
godane |
so i found another show from koreanet |
|
22:31
🔗
|
godane |
here is a url for one of them: mms://vod2.kbs.gscdn.com/koreanet-2/gwangju/2008/tvpro/power/01/02.asf |
|
22:31
🔗
|
pizzaiolo |
nice |
|
22:31
🔗
|
pizzaiolo |
godane: have archivebot get it |
|
22:32
🔗
|
godane |
can archivebot do mms streams |
|
22:32
🔗
|
godane |
? |
|
22:33
🔗
|
godane |
cause i thought i couldn't |
|
22:34
🔗
|
pizzaiolo |
oh yeah I missed that part lol |
|
22:42
🔗
|
|
schbirid has quit IRC (Quit: Leaving) |
|
23:12
🔗
|
joepie91 |
[22:42] <pizzaiolo> I mean it would be pretty undemocratic if they didn't |
|
23:12
🔗
|
joepie91 |
uh, no, not at all |
|
23:12
🔗
|
joepie91 |
in fact, it's undemocratic that they *did* |
|
23:12
🔗
|
joepie91 |
despite clear signs of the referendum result being completely wrong |
|
23:12
🔗
|
joepie91 |
"democracy" isn't "blindly running the numbers": |
|
23:12
🔗
|
pizzaiolo |
to be clear, I'm not defending the brexit result |
|
23:18
🔗
|
|
GE has quit IRC (Quit: zzz) |
|
23:19
🔗
|
|
odemg has quit IRC (Remote host closed the connection) |
|
23:23
🔗
|
|
odemg has joined #archiveteam-bs |
|
23:58
🔗
|
Frogging |
the referendum itself wasn't wrong but they probably should have had a threshold higher than 50% |