Also posting a LinkedIn event photos with people's real names and pictures in a top post on HN along with provocative framing like "partying in Vegas" about this outage is pretty shitty. Atlassian is a big company and has a huge engineering and sales department, you have no idea if any of those people at the Vegas event had anything to contribute with the outage response. For example in your link there are literally people talking about some new mobile app their team is launching at the event, I doubt any of those people are involved in the outage response.
No one posting on HN unless they work at Atlassian in a leadership role is in any position to even start assigning blame, call for firings or publicly shaming people (the later of which you shouldn't be doing even if you 100% knew for a fact that they were at fault).
Do better.
That’s my perspective. Reading your comment it feels like between the lines you don’t think this is as serious as other people. If you have a different perspective, could you come out and say it? Can you elaborate on why a “Global Head Of” isn’t a senior leader? Do you think this is an unfortunate tech outage that is to be expected from any b2b tech company of this size? If you did, I wonder how many people here would disagree with you. Implicitly, the person to whom you are responding does. “Business collapse” and “vegas party” do not look good getting caught in bed together.
Your point would come across much better if it wasn’t mixed with moral outrage. If you have alternative opinions please share them and back them up. Then you will have earned a little more of the massive amount of social capital you need to tell someone, in public and quite rudely, to do better.
My previous company lost customer data. Someone deleted a previous employee's account which surprise contained a production customer website. We didn't halt sales and outreach.
I mean RSA lost all of their encryption seeds in 2011. Every account was compromised. They still do sales today.
whether this is cancel culture not it definitely looks like finding an easy target to shout at
> Atlassian's Global Head of Customer Success probably should have been fired
I don't know who carries the true blame here, but calling for the resignation of a top manager isn't really unreasonable or immoral on the face of it.
The OC is directing us to information made public by Atlassian team members themselves. The OC did not make the information public. Anyone could find that information with little effort.
I’m not really sure why you attribute such intensity to the OC’s rather benign comments.
Can't blame support for something an engineer did.
This was a conference they hosted, not just some Atlassian team members partying in Vegas: https://events.atlassian.com/team22
Thousands of Atlassian customers bought tickets, flights, and hotels for the event. It's unreasonable to suggest that they should cancel it all days before the event.
> Atlassian's Global Head of Customer Success probably should have been fired but here she is promoting Atlassian Cloud on LinkedIn three days ago: https://www.linkedin.com/mwlite/in/gertie-rizzo-5b70061
She responded that it was a scheduled post. This is normal on LinkedIn: Marketers write a lot of posts all at once and schedule them to come out periodically. They're not literally sitting down in front of LinkedIn and typing out their daily puff piece.
Please, let's drop the pitchforks. It's not a good look for us on HN, especially when the facts are being completely misinterpreted.
And the bit about Atlassian employees being in Vegas has nothing to do with anything - as if the entire company is supposed to shut down its planned celebration because of an incident that a small subset of the company should be handling.
These are the kind of incidents where parties and celebrations are put on hold and everyone does what they can to help, regardless of department or title.
They’re talking about the outage lasting for two weeks without any communication as to what’s going on.
But,
You can imagine problems restoring one
individual tenant's data to an otherwise
active database with many tenants
any cross-tenant primary keys
That's such a recipe for disaster; giving each client their own database seems like the easiest win in the world.
But I'm not intimate with Atlassian products. Maybe they have some products where that's not practical for some reason.
1) Its super common even in multitenant systems to have a common database with configuration information (for example) which serves all tenants, and tenant-specific databases used alongside that to host their private data.
2) Back when sharding started to be a popular scaling pattern, tenants were not always split up by the tenant boundary but by some other reliable key. Obviously this isn't true multitenancy and I think most DBAs would discourage the pattern today. However, given the age of the products at Atlassian (and assuming a fast-and-loose engineering culture, which has been alluded to elsewhere) its entirely possible that parts of these products as well as the entire product itself may use this kind of sharding.
Bottom line, we can only hypothesize unless and until someone from Atlassian actually details their architecture (which may have happened? I dunno, I haven't been paying that much attention to it…)
1) Its super common even in multitenant
systems to have a common database with
configuration information (for example)
which serves all tenants, and tenant-specific
databases used alongside that to host their
private data.
One wouldn't imagine that the shared config database would have a dependency on any of the individual client databases and that they could therefore be moved/dropped/restored at will, independently of the shared config database.
2) Back when sharding started to be a popular
scaling pattern, tenants were not always split
up by the tenant boundary but by some other
reliable key.
Nothing, and I mean absolutely nothing, that Atlassian has to offer is rocket surgery-kind of hard... yet, here we are... not being particularly surprised at all.
But the idea of doing a full restore to a new environment, then only enabling accounts for the impacted tenants, is a good one.
They're in control of the architecture: rollback, backup, and recovery should all be considerations
On a single Postgres instance you can (at least theoretically) have 4 billion databases per instance.
In that case, the tradeoff between isolation and ease of development is made. That said, having a schema per user (even if in the same physical database) seems like a nice approach, if you can stomach the overhead and added ops complexity.
And I don't think even pgbouncer would help.
However it doesn’t imply there is vindictive drive
some people will not have a good month career wise, some people will lose trust, some people will be an consequences of regaining the public confidence
> Yes Andy it was and I didn't realize that my scheduled posts were still going out. They have now stopped.
If I was in customer success at an enterprise vendor I doubt I'd be let anywhere near the tools to get this back up and running. These guys are generally in the way rather than helping in a situation like this.
Head of engineering or some product rather than customer support? That might be a different outcome.
When do execs living it up at the fucking Wynn Encore while the house burns down start to not get another job?
They’ll keep pulling this shit until it cost money.
I’m still trying to kickstart a second act years later, because I’m trailer trash and it’s hard work when you’re that.
The most inexusable thing when engineering mistakes happen is something like: failing to bring back people's dead friends and family members.
Not that some customers of some groupware application weren't communicated with for a week.
> Yes Andy it was and I didn't realize that my scheduled posts were still going out. They have now stopped.
I'm still unclear where "Global Head of" for something like this fits in an org chart (Who do they report to? Who reports to them? Is it within a marketing, sales or customer support tree? Etc). Title inflation and whatnot considered...
https://www.linkedin.com/feed/update/urn:li:activity:6918235...
Atlassian famously eschews the exact sales teams whose job it would be to manage direct customer comms in an outage like this one, and to be the lightning rod for the understandable customer frustration. In the past, I've been the guy that gets the angry text message from the customer and has to carefully paper over the gaps in communication from higher ups. It's not fun being the neck that gets choked.
The complete lack of meaningful communication for so long indicates to me that Atlassian doesn't have a meaningful feedback mechanism from the field back up to the executive suite - the exact feedback mechanism that Sales and Sales Engineering teams fill in most SaaS orgs. Customer Success should fill that role, but IME don't have the same incentives, pressure or influence as sales teams watching half their yearly comp go down the pipes.
Also, GP's quote
> Engineering mistakes happen.
I don't like this statement because it offers consolation at the expense of unintentional normalization.
The Negative fall out was not due to the deletion of customer data, as the Story and multiple customers have stated the negative fall out was the SILENCE / lack of communications, which is Sales / Customer Service not engineering
As the comment I was replying to noted while engineering was trying to recover from what might possibly be the biggest outage in the history of the company Sales was partying and not handling customer communications
That (the failure to communicate with customers) should be a resume generating event of all leadership customer service / sales. It will not be because sales will simply redirect their failure on to engineering in the exact same manner you just have
Sometimes manure will always hit the fan. Being robust means being able to handle that.
It's delusional to think software can be flawless in the real world when it's used by an untold amount of people, on all manner of devices, possibly running different OS's with different versions on networks that can be configured all sorts of ways. Thats not to mention all the dependencies involved in creating high level software, from the third party libraries to external services like cloud storage.
You anticipate there will be problems and make sure there are processes in place to manage them when they inevitably occur. Thats the exact opposite of laziness.
Human error is probabilistic, and the probability of making an error cannot be zero.
On the flip side, it’s infeasible to use only provably correct systems; not lazy, but literally not a practical option due to compute costs, developer time, what formal techniques can even be applied to the problem at hand, etc…
Let’s say there’s a 10% chance of any given feature being broken. Write a test, (which has another 10% chance of being broken) and now it’s only broken if the test and the code are broken, and broken in the same way. So we’re down to <1% chance of failure. In my experience most bugs that make it past testing do so because you forgot a test.
Then add a backup / redundancy system. That has a 10% chance of failure, but if you test it regularly then the backup / restore process only has a 1% chance of failure.
Now we have a system that’s pretty reliable in practice, made out of pieces which are only 90% reliable. And no need for PhD level formal methods.
Just do the obvious robustness steps: Write unit tests. Run them with every commit. Have a backup system. Test it. Have redundant servers. Do stages deployments. Monitor your servers and have an on-call roster. Then when everything is working well, add a chaos monkey to increase the failure rate of all of these parts so your team & software gets practice dealing with problems.
The fact that this bug slipped past all of their reliability engineering - past code review and testing into production and in a way they can’t recover - that smells of sloppy work.
The trouble was the restore would set back everyone’s data to that point in time, whereas only some customers data was impacted.
They've underbudgeted for engineering, and they're feeling that now.
You're never going to get perfect error handling in any non-trivial system.
Being robust means that you plan for particular states (like "deleting the production data"). That doesn't mean that your plan is any good, or that your plan will fix the problem, only that you have a sequence of steps developed in advance of the problem.
Sometimes the state in question is considered too unlikely[1] to ever occur, so is ignored with the caveat "too unlikely", such as planning for the case when the company files for bankruptcy and all software needs to be sped up by a factor of two in order to halve computing costs.
Not all possible future states need to be accommodated for in the tech stack - that doesn't mean stack is not "robust".
[1] Or if likely, is such a large problem that all the other problems are irrelevant.
Without knowing Atlassian's org structure all I, an outsider, know is that engineering had an issue and remediation status communication has been lacking. If anyone should take "blame" it should be the bad engineering practices (usually a lack of funding or inability for upper management to prioritize engineering risk) which led to an outage of this magnitude and the status update mechanisms in place. Even then, I question how much better status communication would even get Atlassian given the sheer length of this outage, which is why I think the blame should ultimately land on engineering practices. Hopefully Atlassian learns from this but I'm afraid the damage is done. A SAAS system which takes this long to heal is unacceptable in 2022.
In the contest of this outage, the "Global Head of Customer Success" should absolutely be taking the blame for failing to communicate with their customers.
You can attempt to deflect the blame purely onto engineering side of things, as people in sales often to, but you are objectively wrong.
I have suffered long outages with vendors before, the key for me was always COMMUNICATION. They should be communicating with every customer impacted, they should be telling them on going progress, where in the queue they are, etc etc etc.
Trust me as someone that make purchasing choices and recommendations the length of this outage is not the issue, the lack of communications is. As someone that makes purchasing choices and recommendations I do not blame the engineering, I blame sales or "Customer Success" or what ever PR name you want to further deflect sales to be....
So I don't see why partying at the same time could be an issue. Thats the engineering which made the mistake. So even though they could have communicated better ( and we don't have all the details, we don't know). The true people at fault are Engineering and product leaders.
Engineering mistakes is not an excuse to blame others, and there is difference between a mistake and removing data of production customers
The most inexcusable thing is not communicating with the paying customers who have been affected for over a week.
Atlassian's Global Head of Customer Success probably should have been fired but here she is promoting Atlassian Cloud on LinkedIn three days ago: https://www.linkedin.com/mwlite/in/gertie-rizzo-5b70061
Actually reading a bit more, it seems like their customer team was partying in Las Vegas instead of taking care of business: https://www.linkedin.com/mwlite/feed/hashtag/atlassianteam22
Priorities.