I think it's a staff member. Can't remember first name, Yuri maybe, who is fairly active with the project.

Nope, that would be me.

Tough night dude. I'll buy you a drink or three if you're ever in Sydney...

Alas, poor Yorick!

Sorry for the rough night Yorick. This could happen to all of us but of course it happens to the person that is working the hardest. <3

I'm going to add to your message box unnecessarily, but I want to say I love GitLab and it's a shining example of a transparent company. I still have ambitions to work there someday, and this event is hopefully a net gain in the end, in that everyone here and there learns about backups.

Thanks for the transparency. Doesn't always feel good to have missteps aired in public, but it makes us all a little better as a community to be clear about where mistakes can be made.

I've been there myself, it was at the start of my career and almost ended it. I know how incredibly emotional this kind of thing can be. Just understand you aren't the first, you wont be the last and shit happens. If you are ever in Philly I'll buy you a beer if you drink, and a dinner if you don't.

Was just trying to push earlier today and found out about the issue. Sorry man! Drinks on me in Montevideo, Uruguay. This stuff happens, more than most of us are willing to accept so, here is for your transparency and you know, fix it, learn it and on you go!

Infrastructure ops here. Happens to the best of us. Just have to get back on the horse.

All I could think as I read the postmortem was "I hope YP is doing okay". We've all been there, and I hope you're doing okay!

I'll pour one out for you next time I go out.

Unlucky mate. Even monkeys fall out of trees. Good luck with the fixing.

Dude, we've all been there. You are neither the first nor the last. It's never a single person. One day the technical side of our little startup collectively went to a conference (back then it was only two engineers plus a technical leader) while our server DoS'd itself by broken mail processing... we had a rough night in a Belgium hotel figuring out who attacked the site to realize it was ourselves. The 10k block of missing image IDs always stood as a reference not to leave even a low traffic site unmonitored. It happens.

Hey man, don't beat yourself up over this. It's shitty but you found some flaws in the process, in the setup, and y'all can make things better because of that.

Been there, done that (like most of us that deal with any kinds of infra). Hang in there and best of luck on the recovery efforts!

Yorick! Thank you for the transparency, I know how tough incidents like these can be. Stop the bleeding, figure out how to handle this better in the future, but most of all, take care!

Hey - thanks for GitLab, and my condolences on your (data) loss.

It happens to everyone eventually, so don't feel too bad. But at least now you'll have a much more robust backup process.

We've all been there! Feel better man, it could be a lot worse :)

Go sleep man, shit happens

Always welcome for a drink when you are in Osaka/Tokyo.

You have drinks of your choice in Stockholm too. Man hug.

hugs

Yes, those are the initials of an employee here. Sorry for the confusion!

As much as I appreciate GitLabs extreme openness, that's maybe something that by policy shouldn't be part of published reports. Internal process is one thing, if something goes really bad customers might not be so good at "blameless postmortems" if they have a name to blame.

That is why we went with initials. And I hope customers understand the blame is with all of us, starting with me. Not with the person in the arena. https://twitter.com/sytses/status/826598260831842308

It seems to me that, as a customer, it is blame-shifting away from the company to a particular person. Blameless post-mortems are great, but when speaking to people outside the company I think it is important to own it collectively, "after a second or two we notice we ran it on db1.cluster.gitlab.com, instead of db2.cluster.gitlab.com." I believe this isn't your intention, but that is how I interpreted it.

In our postmortems we explicitly avoid referring to names and only refer to "engineers" or specific teams. There is no reason to refer to specific names if your intention is a systems/process fix.

To me those "Engineers" read as faceless replaceable cogs. This initials make it personal, its better, we can now say "YP" thats exactly you, hey, chin up. Sounds better than "engineering team 42".

You write CEOs name on all your publications, of course always taking credit/glory, but why not let engineers do the same, take credit/ownership when doing a nice commits, and when fucking up. We're all people first, and prefer to speak/talk to people and not Engineering Team MailBox at Enterprise Corporation.

We are all responsible for this, starting with me. We don't blame any person. References to people are just to determine what happend. Also see https://twitter.com/sytses/status/826598260831842308

Yeah, I know, I read your reply. I think it is wrong.

Is your username a Spin reference?

Haha, it wasn't intentional. I'm just a space nerd. That book ranks pretty highly on my list of things every space nerd should read though.

I quite enjoyed it! Also, +1 for space. And Greek.

From looking at the context of the way YP is referenced (link to a slack archive), I believe YP is an employee.