Re:The GitHub Availability Report
by hn_throwaway_99 on Thursday July 16, 2020 @12:31PM
(#23860625)
> I have started looking at tying together clusters of business apps that each have independent SQLite datastores using an application-level protocol for replication & persistence (over public HTTPS). This allows for really flexible schemes which can vary based upon the entity being transacted. With hosted SQL offerings, you are typically stuck with a fairly chunky grain for replication & transactions. If you DIY, you can control everything down to the most specific detail.
Ugh, I'm totally going to sympathize with whoever eventually joins your company after you've left who has to own this.
I don't think I have provided enough contextual details in order for this type of conclusion to be reached.
I'd be happy to offer a hypothetical if you are interested in debating the merits of my approach.
I'm not sure it even makes sense for smaller platforms any more. The metric of most business success is really growth. Tying yourself to something which is troublesome to grow because it favours doing things that are difficult to scale is an evolutionary dead end. I've spent the last 25 years constantly battling this problem in relational databases. Moore's Law has been pretty kind so far assuming you have the money to buy ridiculous computers and licenses.
I am out of touch of what is available in the NoSQL side of things but about 5 years ago there were too many things which had a terrible failure mode somewhere in the stack thus I'm never sure which way to go instead.
Boy in 2020 I can't see describing databases as difficult to scale. Expensive, sure, but these are solved problems, and the tradeoffs for going the NoSQL route will likely exceed its perceived benefits.
Modern relational databases are great at what they do.
They are until they're not, at which point it's too late. I deal with things teetering on the edge.
At some point you get to machine sizes that won't fit in AWS as an example.
That's true of, well, anything. A non-RDS / NoSQL datastore doesn't save you here!
I have started looking at tying together clusters of business apps that each have independent SQLite datastores using an application-level protocol for replication & persistence (over public HTTPS). This allows for really flexible schemes which can vary based upon the entity being transacted. With hosted SQL offerings, you are typically stuck with a fairly chunky grain for replication & transactions. If you DIY, you can control everything down to the most specific detail.
For instance, when going to persist a business object's changes, you can have a policy configured that reflects over the type and determines how many nodes need to be replicated to and which ones should replicate synchronously vs asynchronously based upon it. Log entries may only be async to 2 nodes, but you may decide that any accounting transactions should be replicated to 2 near nodes synchronously, with 2 more far nodes being async.
Additionally, you could inspect various business facts in the entity (or session related to the transaction) in order to determine ideal persistence strategy. One powerful example here could be to use a user's zip code to determine the geographically-ideal node to store the primary replica of their account & profile data. This could allow for lower latency access for that specific user.