If you are running an app that is basically a UI layer over a database, where would you expect significant failures to occur? The users are already debugging possible problems with git locally by simply using it, so UI-database interactions are where all the action happens on products like github.

When I was CTO in some companies we had very different problems, with DBs being only one of them (and thankfully not four hour long outages in two months).

I might be wrong, but you probably didn't handle as much traffic as Github does. Bugs and outages probability is a asymptotic to normalized traffic. At least intuitively.

No we didn't luckily but we also didn't have that much money. Biggest problem was 50k+ people trying to reserve a limited amount of stuff at the same time and 5k+ logins/second (at first we made the mistake to write last_login to MySQL ;-)

We changed to a write behind setup (e.g. with Redis PubSub or any other queue). I think everyone with some experience will use async write behind and not sync writes during a request as we did.

How did you go about saving login/device data? We are going to do this soon in our app and I'm looking for good use cases and solutions

What confuses me is that CPU starvation caused crashes. Sure, too few CPU ressources should cause a slow down but the database process itself shouldn't crash as a result. I haven't dealt with MySQL that much but at least for Postgres, I've never seen 100% load to cause a node to fail.

I could see a buffer for requests filling up too much so that it causes a no memory crash.

But even that seems preventable.

OOM issues often don't _really_ cause a crash unless you are running with real memory only. Usually long before you are out of total memory you are using so much virtual memory that the kernel is thrashing IO to try keep up with all the page faults. At this point the machine hasn't crashed, but from the outside it might as well have because it can't respond to anything new, even diagnostic intervention, in a timely fashion.

CPU overloading, without memory starvation, can have a similar effect, though that doesn't tend to fall off the same efficiency wall as when IO is the issue (unless something has literally kicked off a fork bomb) because of the orders of magnitude differences in latency and work throughput.

A swap livelock would be worse than a crash, you really don't want a DB backend to start swapping in this kind of application.

No, but from a diagnostics PoV it can be important to know the difference between failure modes that, unless you are physically near the device so can see drive lights flickering (or hear heads moving if using traditional drives), could otherwise look identical.

I think the hosted SQL model may be fundamentally flawed when you are trying to do something as complex as what GitHub is doing. You are basically saying "Here, MySQL/Postgres/Oracle/et.al., I trust you with 100% of my replication & persistence logic. Good luck optimizing all that SQL!".

I have started looking at tying together clusters of business apps that each have independent SQLite datastores using an application-level protocol for replication & persistence (over public HTTPS). This allows for really flexible schemes which can vary based upon the entity being transacted. With hosted SQL offerings, you are typically stuck with a fairly chunky grain for replication & transactions. If you DIY, you can control everything down to the most specific detail.

For instance, when going to persist a business object's changes, you can have a policy configured that reflects over the type and determines how many nodes need to be replicated to and which ones should replicate synchronously vs asynchronously based upon it. Log entries may only be async to 2 nodes, but you may decide that any accounting transactions should be replicated to 2 near nodes synchronously, with 2 more far nodes being async.

Additionally, you could inspect various business facts in the entity (or session related to the transaction) in order to determine ideal persistence strategy. One powerful example here could be to use a user's zip code to determine the geographically-ideal node to store the primary replica of their account & profile data. This could allow for lower latency access for that specific user.

> I have started looking at tying together clusters of business apps that each have independent SQLite datastores using an application-level protocol for replication & persistence (over public HTTPS). This allows for really flexible schemes which can vary based upon the entity being transacted. With hosted SQL offerings, you are typically stuck with a fairly chunky grain for replication & transactions. If you DIY, you can control everything down to the most specific detail.

Ugh, I'm totally going to sympathize with whoever eventually joins your company after you've left who has to own this.

I don't think I have provided enough contextual details in order for this type of conclusion to be reached.

I'd be happy to offer a hypothetical if you are interested in debating the merits of my approach.

I'm not sure it even makes sense for smaller platforms any more. The metric of most business success is really growth. Tying yourself to something which is troublesome to grow because it favours doing things that are difficult to scale is an evolutionary dead end. I've spent the last 25 years constantly battling this problem in relational databases. Moore's Law has been pretty kind so far assuming you have the money to buy ridiculous computers and licenses.

I am out of touch of what is available in the NoSQL side of things but about 5 years ago there were too many things which had a terrible failure mode somewhere in the stack thus I'm never sure which way to go instead.

Boy in 2020 I can't see describing databases as difficult to scale. Expensive, sure, but these are solved problems, and the tradeoffs for going the NoSQL route will likely exceed its perceived benefits.

Modern relational databases are great at what they do.

They are until they're not, at which point it's too late. I deal with things teetering on the edge.

At some point you get to machine sizes that won't fit in AWS as an example.

That's true of, well, anything. A non-RDS / NoSQL datastore doesn't save you here!

It's more about the granularity of the architecture than it is any specific technology used within it.

Three out of four times MySQL was involved in the success of products and companies.