You can reverse this as 'how much extra complication comes from having many disparate databases, so now you push all ACID complications at the company level instead of the software(rdbms)'
It is a tricky problem!
Having a single source of truth is a tremendous simplification, especially for the people.
It should be ideal that the other databases are pure materializations on top of it, but then not matter what you have the need to input more stuff that is not part of the central one, and now you have both problems.
Using the language of 'software architecture: thee hard parts' this forces your entire system into a single 'architectural quanta', basically it is a monolith.
There are situations where monoliths are acceptable or even the least worst option.
The 'Fallacies of distributed computing' covers most of these.
It can work better if you use a hierarchical model, but OLTP is almost exclusively the relational model.
ACID transactions become pretty brittle, software becomes set in stone etc...
There are use cases, but the value proposition is typically much weaker
Oddly, I'd argue that the better way to view this is that the central database is the materialization of the data sources scattered around where work actually happens. As such, you have to have constructs that account for provenance and timing introduced there that aren't as necessary at the edges.
If you could elaborate on this further, I and others are probably very interested in reading more about it.
Briefly looking, I can't find the books that I thought covered a lot of this in a good way. Will keep looking, apologies.
I can't recommend it highly enough.
[1] https://www.amazon.com/Designing-Data-Intensive-Applications...
- OLAP: most queries need most records (aggregations span large swaths of data)
- OLTP: most queries access just a few records
Also, in OLAP, in many cases, you can live with a single-updater model without much trouble, where OLTP, the strength is to have many concurrent updaters (but mostly non-overlapping).
- OLTP: write-mostly, index-seek heavy, ~all queries pre-defined up front
OLTP - You are dealing with the ins and outs of people using stuff to do things. You buy something, you check out something, you some way perturb the state of things. This should not require large amount of row lookups in 99.9% of cases.
However, at global/twitter scale, just basic CRUD operations become pretty hard for OLTP, so scalable OLTP in the cloud is a pretty big deal, especially when you get fun things like phantom read/writes as writes may take time to get to distributed read replicas, so a user may send a tweet and then not see it on their profile for a while.
But global/twitter/reddit etc. scale does matter in that I don't really remember a PHPBB or slashdot or HN swallowing my comments, ever. Even if you get an error you can always just go back in the browser and your input box is there with your comment.
But reddit has been effing atrocious over the last few months. You get "Internal Server Error" when trying to do any voting or commenting a lot and it won't go away until you refresh the page like 17 times (until presumably you round robin get to some BE server that actually works. It's also been swallowing comments, where about every third or fourth comment it accepts the comment instead of throwing an Internal Server Error at you (which would be preferable) and instead it "accepts" it but it will never show up. Ever. Only chance is to copy every comment before submitting, in case you have to re-submit it from scratch.
I think I agree across the board. HN and Stack Overflow are both amusing examples of much simpler architectures that kind of get to the point, though? Such that I'm not clear if we are on the same page, there.
(And I, sadly, have zero experience with Reddit.)
Really? I remember it happening a lot.
> Even if you get an error you can always just go back in the browser and your input box is there with your comment.
That's a relatively new feature and doesn't always work. (Old, I can't speak to new) Reddit's "push a button and it submits without moving off the page" is much nicer.
The problematic case is that very often now it accepts your comment without any errors and... Then nothing. Your comment will never ever show up. The system swallowed it. Whole. Without chewing.
Huh. I've never seen that. Does it not show on your user page even? I've known it to take a few seconds for comments to show up, and mods on certain subs love to silently delete comments (sometimes automatically), but I've never seen them just disappear.
I do also have the case where it disappears but a refresh / going to the user page will show it, i.e. something async on their end was just a few milliseconds too slow or something.
And posting the exact same comment again (I c+p them now) works. So it wouldn't be any automatic removals and it's too fast for manual removals.
https://www.infoq.com/presentations/reddit-architecture-evol...
That and the Instagram architecture video are such eye-opening examples that I always send to juniors when they have doubts about their own abilities. In most cases, we're astronomically no where near Reddit/Twitter/Instagram scale, and yet we avoid a lot of the common pitfalls we see presented in those videos. Side note, I'm shocked at the quality, but hey "Time To Market!"
The Instagram video: https://www.youtube.com/watch?v=hnpzNAPiC0E
I had the same eye opening experience when I first "met" Jira. I was pretty fresh out of uni myself, yet I knew you should never use a business key that people might want to change and always use surrogate keys for your database.
And here is Jira "happily" using Issue Key as both the business key that signifies which project an issue belongs to, which can very obviously change as soon as you move a ticket to another project and as the primary key in their database and thus for everything related to an issue like say issue links.
I was laughing so hard that you couldn't tell I was also crying and cursing at the same time coz I had to deal with that fact.
They've since introduced ids for things but ... duuuudes! WTF! lol
From my experience, the vast majority of complications in systems is people not realizing they are asking an OLAP question while wanting parts of OLTP semantics.
I'm also curious how much of this extra complication comes from having a central database that tries to be a source of truth for everything? As an easy example, inventory systems that try and have a cloud source of truth have obvious problems when on the ground inventory issues crop up. That is, a lot of the complication is between the distributed nature of the application and database, sure. But, another large source is the non-distributed abstraction that a central database can represent not centralized activity.