News for hackers. Stuff that matters.
Re:Inside the longest Atlassian outage by abraae on Wednesday April 13, 2022 @05:02PM (#31020218)
This is extremely poor for a large SaaS company.

A standard RFP question for SaaS should be:

- Can you restore data for a single customer, and if so, what is the RTO for that operation?

A smaller SaaS could be excused for only thinking about full database restores. When you're a scrappy upstart, thinking about hypotheticals is less important than survival.

But for any decent size multi-tenanted SaaS, it's imperative that you have the ability to selectively restore individual customers.

The usual approach is to do a full database restore into a separate instance, then run your pre-prepared "restore customer" scripts to extract a single customer's data from there and pump it across your prod instance. In Oracle for example you might use database links to give your restore code access to prod and also the restore instance at the same time.

Atlassian - MUST DO BETTER.

Re:Inside the longest Atlassian outage by Animats on Wednesday April 13, 2022 @05:45PM (#31020695)
Rather, customers must stop using Atlassian cloud services.
Re:Inside the longest Atlassian outage by imroot on Wednesday April 13, 2022 @06:31PM (#31021121)
Which is becoming more and more difficult due to them focusing on Cloud Products (my on-prem renewal jumped almost 8x this year).

I’d rather use request tracker or bugzilla over Atlassian these days

Re:Inside the longest Atlassian outage by randmeerkat on Wednesday April 13, 2022 @11:53PM (#31023242)
Try Kitemaker, it’s a YComb backed JIRA alternative.

https://kitemaker.co/

Re:Inside the longest Atlassian outage by Aeolun on Thursday April 14, 2022 @01:19AM (#31023651)
This landing page is exceedingly light on explaining what problem they solve for me.
Re:Inside the longest Atlassian outage by Karrot_Kream on Wednesday April 13, 2022 @07:37PM (#31021649)
This is a great opportunity to be an Atlassian competitor. I'm sure more than one business added Atlassian cloud services as a business risk in the meantime, even if they weren't affected.
Re:Inside the longest Atlassian outage by galangalalgol on Thursday April 14, 2022 @12:20AM (#31023365)
gitlab seems to be eating their lunch already. Its bit bucket plus Jenkins plus the best bits of jira and it's starting price is free... I like confluence a lot, but separate design/documentation tools with their files pushed up to the repo are good enough. Latex, doxygen, or PowerPoint, whatever you want.
Re:Inside the longest Atlassian outage by dx034 on Thursday April 14, 2022 @03:11AM (#31024169)
I still don't get why they didn't separate clients on a database level. Sure, put many clients on one database server to save resources. But why not use different databases? They cost nothing and provide perfect separation. It also drastically lowers the attack surface as you can set all permissions via database software. And if they had done that, this would've never been a multi-day outage.

If Jira was a product used by individuals I'd get it. Maybe a database is overkill for a sole developer. But pretty much all users of Jira are companies with tens or hundreds of users on average. I don't see how separating on a db level is overkill in that situation.

Re:Inside the longest Atlassian outage by mustardo on Thursday April 14, 2022 @07:32AM (#31025477)
Using separate databases or schemas per tenant comes with the following problems

* Managing schema migrations across every DB

* You cant query across the DB, want to know some cross tenant thing for ops? That's now a lot harder

* Connection pooling and resource usage can be harder to manage

Most systems I've worked on use a single DB with a `tenant_id` col on every relevant table, it's easy to have your query builder slap in the auth'd tenant I'd. This approach does come with issues like saving and restoring an individual tenants data

Like a lot of things in life, it's a trade off

Re:Inside the longest Atlassian outage by donavanm on Thursday April 14, 2022 @03:29AM (#31024258)
> why not use different databases? They cost nothing and provide perfect separation.

I understand the sentiment, but This is a pretty simplistic take that I very much doubt will hold true for meaningful traffic. Many databases have licensing considerations that arent amenable. Beyond that you get in to density and resource problems as simple as IO, processes, threads etc. But most of all theres the time and effort burden in supporting migrations, schema updates, etc.

Yes layered logical separation is a really good idea. Its also really expensive once you start dealing with organic growth and a meaningful number of discrete customers.

Disclaimer: Principal at AWS who was helped build and run services with both multi tenant and single tenant architectures.

Re:Inside the longest Atlassian outage by dx034 on Thursday April 14, 2022 @03:35AM (#31024289)
Don't you usually license based on server resources? Or do you know really have to pay per database/schema? At least on-prem licenses tend to be based on resource usage, not on the number of databases or schemas. I'm not talking about different db processes, just databases/schemas within a database.

And for migrations and schema updates I'd see this as a huge advantage. Migrating customers one by one is much easier than everyone at once. You also never have the issue that operations at one customer could cause a global lock affecting other customers.

Of course resource sharing isn't easy in this scenario, but you'd never want to connect data between customers anyway so I don't see the issue with that.

But maybe it works harder in a cloud environment where more is abstracted away.

Re:Inside the longest Atlassian outage by donavanm on Thursday April 14, 2022 @06:44AM (#31025163)
Ah, when you said "database" I assumed you meant a dedicated single tenant instance of an RDMBS (or similar), and not necessarily something like dedicated tables. I will admit to being a decade out of touch with the vagaries of "processor", server, and client access licensing. In my relevant past I've only worried about (RDS/EMR/Redshift/etc) instances and tables.

Very fair call out on having more granular, discrete, instances for things like DML/schema updates and expensive queries. I love fault isolation and have had many sad days oncall when we exceeded the capabilities of The Database.

I wouldnt say it's harder because it's more abstract. I think the general motivation is to desperately avoid anything that scales cost/effort with the number of users. Even if it's sublinear a team can really drown under the cost of scaling up a service. And that's a serious consideration when a baseline expectation is to go from 0 to 10,000 or 50,000 active customers in just a few years. The care and feeding of (for example) 10 multi tenant partitions is just simpler than having to monitor & operate 10,000 independent databases with wildly divergent usage profiles. I will grant this hyper growth is not a common scenario for the industry, or if it is then its "one of them good problems."

I'd also say I have worked on a project that did have independent data tables for each customer instance. And we spent a meaningful amount of time abstracting away table creation/migration/etc, a common DAL that abstracted away the multitude of tables, common monitoring, etc. It has made some things around data migration & management easier but I honestly don't know if it's more efficient than multi tenant clusters in the long term. But the only way the economics and operational effort has worked is by going "all in" on using "serverless" technologies that efficiently scale to zero and have no carrying cost when idle

Re:Inside the longest Atlassian outage by abraae on Thursday April 14, 2022 @03:24AM (#31024229)
There are lots of downsides to doing absolute partitioning of tenants (along with lots of upsides as you point out).

Really annoying things that slash your velocity. You can't easily run pan-customer queries, can't aggregate data.

Ironically too using a single database makes full backup and restore much easier.

Re:Inside the longest Atlassian outage by dx034 on Thursday April 14, 2022 @04:34AM (#31024557)
Wouldn't you extract data anyway to another system for analytics? Running analytics queries on production databases seems a bit risky in any setting?
Re:Inside the longest Atlassian outage by treis on Thursday April 14, 2022 @01:32PM (#31030269)
You can have a read-only replica if you're concerned about performance.
Re:Inside the longest Atlassian outage by scottlamb on Wednesday April 13, 2022 @06:31PM (#31021117)
Is it standard for a RFP to have a long list of questions like this? I've never been involved in an RFP from either side.

Is it standard to (in addition or instead) to have something more general/forward-looking like: how do you watch other providers' postmortems and apply the lessons to your own system?

> - Can you restore data for a single customer, and if so, what is the RTO for that operation?

If I were to aim something at this specifically, it'd be: can you restore data for N customers or N% of customers, and if so, what is the RTO for that operation?

I mentioned in another comment that Gmail had a similar outage in which they had to restore from tape. https://news.ycombinator.com/item?id=31017160 They had a tool for restoring a single account but not for restoring N accounts in bulk, which would be significantly more efficient than doing the one-account process N times. (E.g., in the case of tape backups, imagine the difference between pulling data from the tape library sequentially for each user vs all N at once, particularly when one tape may hold data for many of these customers.)

Re:Inside the longest Atlassian outage by taude on Wednesday April 13, 2022 @06:36PM (#31021177)
Yes, pages of them. Multiple pages of security questions, ciphers used, how data is stored, when is it encrypted, etc. I filled out a 20 pager once. As the company got better and more mature, we had a bunch of canned answers to make it easier and faster....
Re:Inside the longest Atlassian outage by fphhotchips on Wednesday April 13, 2022 @07:13PM (#31021477)
Entire (excellent) start-ups exist to fill the role of 'RFP library' so that you don't have the whole sales team rewriting the same answers 100 times a year. Loopio saved me hours in the last role I was in that had them - even if you do have to edit some of the responses from colleagues you're not sure passed 9th grade English.
Re:Inside the longest Atlassian outage by porker on Thursday April 14, 2022 @02:18AM (#31023912)
Any other startups you can recommend? I'm filling in my first RFP in a decade and answering what they mean to ask with the questions rather than answering questions literally is not something that comes easily to me.

Plus coming up with an answer to the vague question on "describe your project methodology" (I build what you want, it works - nope, they expect half a page). Or the 3 questions on project management systems and communication software choices that to my reading should have the same answer.

Re:Inside the longest Atlassian outage by fphhotchips on Thursday April 14, 2022 @08:35AM (#31026079)
Not really to be honest. If you've got an RFP to respond to right now then it's a little too late.

That said, I've never had an issue copy pasting the answers to similar questions. As long as you answer the question with the answer!

Re:Inside the longest Atlassian outage by alias_neo on Thursday April 14, 2022 @03:56AM (#31024380)
It's very common to have several pages of questions like this with particular customers, and though they may often come in to sales and marketing people, they'll contain highly technical or operational questions relating to a variety of things such as security algorithms, programming languages (type safety etc) all the way to disaster recovery.

Regarding bulk restore, a big customer doesn't care if you can restore all of your customers' data, they care if you can restore _their_ data, and fast, hence the question of "can you restore data for a single customer?".

Re:Inside the longest Atlassian outage by scottlamb on Thursday April 14, 2022 @09:57AM (#31027080)
> Regarding bulk restore, a big customer doesn't care if you can restore all of your customers' data, they care if you can restore _their_ data, and fast, hence the question of "can you restore data for a single customer?"

This outage should convince them to care. The problem isn't that Atlassian can't restore a single customer—there are people reporting that they've been restored. [1] It's that Atlassian can't restore 400 customers efficiently. So unless the RfP also has a question "will I be first on the list?" and the answer is yes, single customer restore is the wrong scenario.

[1] https://news.ycombinator.com/item?id=31023163 says "I was down, my instance is fully restored right now. ... Not sure what to tell you other than they are fixing life saving companies first, then the rest. That is what they have told us."

Re:Inside the longest Atlassian outage by jamesfinlayson on Wednesday April 13, 2022 @07:49PM (#31021750)
I work on a couple of reasonably small products but generally government customers have a long list of questions (mostly but not entirely about security).
Re:Inside the longest Atlassian outage by drsim on Wednesday April 13, 2022 @05:16PM (#31020394)
Which SaaS platforms provide account-level restores?

If you contact them and say "please restore our data to as it was last week" those I know do not offer this.

Re:Inside the longest Atlassian outage by ibejoeb on Wednesday April 13, 2022 @05:50PM (#31020739)
I did. It was an first-principles architectural decision. A client could request any point-in-time within the contracted period, and it could be either a restoration or a fully operational, parallel instance of the account.

It was initially a cover-my-own-ass design, but it turned out to be an extremely popular feature that was never even used for disaster recovery. Instead, it was used for audit support, trial scenarios, projections, and all kinds of other stuff.

Re:Inside the longest Atlassian outage by jimbokun on Wednesday April 13, 2022 @10:10PM (#31022689)
What kind of data volumes were you dealing with?
Re:Inside the longest Atlassian outage by boardwaalk on Wednesday April 13, 2022 @05:30PM (#31020552)
I wouldn’t expect them to advertise such a thing, but the question is “can they recover from their own mistakes” not “can they recover from mine.” I don’t care if this is with an “account-level restore” or whatever; it shouldn’t be my concern.
Re:Inside the longest Atlassian outage by brazzledazzle on Wednesday April 13, 2022 @09:18PM (#31022349)
I’ve seen customer and resource level restores deprioritized more than once and the only hypothetical given serious thought is avoiding helping customers who accidentally deleted something because of the support burden/cost. No one seems to have much concern for what happens when they’re the ones that screwed up.
Re:Inside the longest Atlassian outage by twistedpair on Wednesday April 13, 2022 @07:51PM (#31021772)
I know plenty of places (small/med startups) with "undelete" and "restore" account/data options built into their admin panels. Engineers shouldn't be flipping bits by hand, under duress.

I really wonder what these Altassian restore tools look like it if takes "hundreds of engineers across the company" to restore 400 accounts. Are backups siloed across many teams?

Re:Inside the longest Atlassian outage by kgeist on Wednesday April 13, 2022 @07:17PM (#31021501)
>Which SaaS platforms provide account-level restores?

We restore deleted accounts on request sometimes. There was a client, for example, who forgot to renew the subscription and did nothing for 30 days, so their account was automatically deleted. We restored it from backups. It helps that every tenant has their own isolated database, so it's mostly a matter of restoring that one single DB. Some microservices store data without DB-level sharding, so we have a script which is able to make a partial dump for a specific account.

There's also a popular option to restore deleted data - nothing is ever hard-deleted (it's marked deleted but stays in the DB) and we have a script which can restore individual records (and related records). There's maybe 5 such requests per month.

We don't offer rolling everything back to a specific point in time, though. Technically it's possible by undoing the event queue but it's untested.

We also have a script to migrate customers from cloud to on-premises and back.

Re:Inside the longest Atlassian outage by Tobani on Wednesday April 13, 2022 @06:02PM (#31020849)
I accidentally built out this feature at a company once and it totally saved our asses a week later.
Re:Inside the longest Atlassian outage by hotpotamus on Wednesday April 13, 2022 @05:22PM (#31020450)
I actually did this once with Dropbox, though it wasn't a feature they actually published. I clobbered my Dropbox directory accidentally, but I was able to find a script someone wrote to roll it back to a previous point in time and it worked quite well. After that I also took my own snapshots just in case.
Re:Inside the longest Atlassian outage by MapleWalnut on Wednesday April 13, 2022 @05:28PM (#31020523)
Dropbox support can rollback your Dropbox account to a previous point in time too.
Re:Inside the longest Atlassian outage by fknorangesite on Wednesday April 13, 2022 @05:31PM (#31020557)
I wouldn't expect it if I just asked. I think it's reasonable as part of their disaster recovery though.
Re:Inside the longest Atlassian outage by shp0ngle on Wednesday April 13, 2022 @11:19PM (#31023091)
> Atlassian - MUST DO BETTER.

it’s not like people will stop using jira and confluence, lol

they basically have a monopoly there

Re:Inside the longest Atlassian outage by bastardoperator on Thursday April 14, 2022 @01:21AM (#31023664)
Nobody likes using this stuff though. I think the new github boards might give jira a run for their money given the cost... free.
Re:Inside the longest Atlassian outage by shp0ngle on Thursday April 14, 2022 @06:21AM (#31025043)
Once a project gets to feature parity with atlassian, it will become atlassian.

Same with SAP. Once your ERP has same feature parity with SAP, it will become SAP.

JIRA is SAP of engineering... yeah why I haven't thought of that before.

Re:Inside the longest Atlassian outage by eddy_chan on Thursday April 14, 2022 @02:40AM (#31024007)
My engineering team runs off Github Issues > Projects > Project Boards but it's been hard to get anyone outside of product managers and devs themselves to be fully immersed in it. Sales and management throw their hands up in the air and say 'Github Issues is too hard'. I'm a PM so I've fought and won this battle to stay on Github Issues because I could quantify that a 5-10% loss in dev productivity in moving to a 'management friendly' tool would equal so many thousands of dollars of developer hours per month.
Re:Inside the longest Atlassian outage by pjmlp on Thursday April 14, 2022 @01:57AM (#31023818)
Here is someone that likes it, and is usually put down by lack of features in competing products.