Inside the longest Atlassian outage

Re:Inside the longest Atlassian outage by jjice on Wednesday April 13, 2022 @03:44PM (#31019308)

Interesting architecture. From a design point of view, I like the idea of full isolation. From an infrastructure point of view I'm a little scared. I'd assume it's actually not that bad and there's a good way to manage the individual DBs and scale them individually.

Really interested if you can share any details.

Edit: I know each wiki is on a subdomain. Does each wiki also have it's own server?

[ Parent | on HN ]

Re:Inside the longest Atlassian outage by kikki on Wednesday April 13, 2022 @04:10PM (#31019654)

There are _many_ databases on each server, last I checked there was around 8 servers (or: "clusters") - and we have it so the traffic is somewhat evenly distributed across each server. There are reasonable capacity limits, and when servers get full we spin up a new one and start accepting new wikis there. I am not in OPS, and they do a lot of work behind the scenes to make this all run smoothly - but from an eng perspective we rarely have issues with this at scale.

Some of this was open source before we unified all of our wiki products, which has a lot of the selection / db logic, at https://github.com/Wikia/app.

[ Parent | on HN ]

Re:Inside the longest Atlassian outage by spookthesunset on Wednesday April 13, 2022 @05:20PM (#31020433)

How do you update the schema on 300,000 databases?

[ Parent | on HN ]

Re:Inside the longest Atlassian outage by kikki on Wednesday April 13, 2022 @07:33PM (#31021627)

It doesn't change often, if we do we just have large automated rollout plans - but we've done mass changes enough times there are good procedures around large DB migrations.

[ Parent | on HN ]