When I talked to my dad about RDBMS he was like "weeeelll, sure, there's things like DB2 UDB that do relational but performance wise, nothing beats reading the data straight by key in exactly the format you need.".
DB2 UDB: "Initial release: 1987; 34 years ago". I.e. what he would rather use and is sort of the NoSQL equivalent is _even older than that_.
Background: he started off with 360 assembler and worked all his working life on IBM Mainframes and the various technologies in and around it. They had it all and they had it before it came to "us". We're just re-inventing most of these things on much cheaper and more open hardware and software.
There's also the argument that businesses hoping to scale might be better off assuming they will need it to save themselves work down the road.
Your Data Isn't That Big https://www.chrisstucchio.com/blog/2013/hadoop_hatred.html
This is an extremely important lesson. You don't have Big Data, unless you're one of a handful of huge companies.
Years ago I joined a mid-size public company and inherited one particular service. Management was adamant that it had to migrate away from MySQL onto Cassandra due to data volume. But there were no metrics being tracked, it was driven by "we have to do it because everyone is doing it".
I started tracking actual usage, growth and performance and it didn't seem to point to a problem that needed solving. So I halted that project (to their credit, they listened to engineering input so I was able to).
A few years later when I left, it was still perfectly fine on MySQL. Nowhere close to running out of capacity. Today it's been almost ten years and while I'm not there, what I hear is that it's still fine.
My point is that large companies that do need to scale hire many developers. FAANG likely hire north of several hundred thousand engineers alone, and according to this article[1] there are only something like 1.5 million software engineers in the US.
[1]: https://www.daxx.com/blog/development-trends/number-software...
Of course that's a oversimplification, but the data generated today is mind-boggling large compared to the 90s.
In most cases, you have colossally fewer than 1M writes per second, and you usually distribute them over several boxes. the IoT avalanche of writes in particular is naturally sharded, that is, all these writes can occur independently, and then you can replicate statistics into an analytical RDBMS.
It's a large write load that needs to be transactionally ordered, like a stream of purchases in a large online store, that can be a hard(er) problem for an RDBMS.
My example was just one use-case where one should consider using a more unconventional DBs like Cassandra or HBase.
For what percentage of companies would you say your statement is true?
The main issue with object relational as the "main" database is its hard to run reports on. Reports (for biz) is half the reason the system exists. I'm not saying ORDBMS don't have a place, but RDBMS is much better suited for real-time data analysis / reports. I think ORDBMS is popular because it's easy for non-database people (who don't know 3rd normal form or SQL) to store stuff in.
Things like fuse also show that it's easy to view other databases as filesystems, too. As a silly example, you can 'explode' /etc/passwd into individual files per entry. Or you could expose even a relational database as a filesystem.
For the latter, you could do something simple and expose tables as directories and rows as files. Or you could do something dynamic, and eg represent (read-only or idempotent) queries as file-names, and their results as file contents.
That wouldn't be so much a traditional filesystem, but would be accessible via a filesystem interface.
For reporting, unstructured data is actually not bad. If your report requires you to scan your whole database, then SQL won't necessarily be faster, you have to think about disk performance as well as how complex of a join(s) do you need.
Nobody has just one app forever. Any org that gets big enough is going to be uncertain about how many apps depend on a particular database, much less whether they’re all in the same language and bundle the same schema rules. Customer service builds weird support tools, accounting and compliance build weird reports …
That assumption is only true if the DB is continually used by different applications. The reality is more likely a large number of applications used by a relatively small set of users, so most of the time there won't be any application writing, and it would be extremely rare for multiple applications to write at the same time.
In reality sometimes access via an API is way too slow or is lacking flexibility.
And I don't get this: "(e.g. if you have multiple applications writing to your database in transactions then you're virtually guaranteed to get deadlocks)"
Isn't this why transactions exist in the first place?
While MVCC avoids deadlocks, you'll still get a lot of failed transactions if they all operate on the same data at the same time, which you'll have to retry, but I'll take that over data inconsistency
(But indeed, with uncoordinated development, deadlocks can occur. Still better than data corruption.)
So, it might be adequate for a small and carefully maintained project, likely like many alternatives.
Or do you use a high-load, distributed, replicated / sharded Mongo configuration? Then I'd gladly listen to more details.
EDIT: to add ... and with HA/DR and where you keep all the good guarantees, queries etc.?
If you're interested, CRDB writes a lot about how they do their transactions on their blog, it's good to know what guarantees you actually get with your DB's transactions.
If all you need is to alleviate the high read load (many selects), nearly every SQL database (even SQLite!) has a free and supported way to create read replicas, usually out of the box.
If you need distributed transactional updates, MySQL / MariaDB has Galera (GPL), and Postgres has Citus (proprietary with a few AGPL parts).
And, of course, there is CockroachDB that prioritizes reliability over speed, but is truly distributed out of the box.
With something like Cassandra you get read after write guarantees that I think you don't get with read replicas and SQL databases but I'm not too familiar with the latest there. Also once you start breaking up your SQL databases into shards you lose all SQL goodness across shards (like joins and other properties).
So no free lunches basically. I'd love for there to be something fast, scalable, queryable, transactional etc. etc. Honestly I haven't looked in a while but every open source SQL solution I've seen in the past that claimed to be durable/reliable/scalable doesn't actually do that under failure modes like NoSQL does and pretty much loses either performance or other guarantees or both. Ofcourse NoSQL doesn't solve some of these problems, but at least you know what you're getting into ;)
With that, you let go of the global ordering, because global ordering / single-point serialization just becomes technically infeasible. At that point you usually don't exactly need them, though. You only need local order within a transaction.
Not reliably. All the different DBs have subtle misbehaviours and mismatches with the tools.
> Performance is actually good and consistent.
Lol no. Indexed joins and unindexed joins look exactly the same but have vastly different performance characteristics. And even if you have a query that currently performs well, you never know when or why the query planner will change its mind.
> Your data is also consistent and respects acid principles.
Only because you drop data on the floor if it's inconsistent, which is rarely a good enough solution in the real world. E.g. someone tries to add a customer but the city has been deleted in the meantime -> they get an error, and the system throws away everything they input about the new customer. You end up having to implement your own dead letter queues etc. and do all the work you would with a NoSQL datastore.
> With proper cashing SQL is as performant when done right and the profiling tools are better.
Caching makes your performance even more arbitrary and incomprehensible - it works great right until it doesn't. Profiling is great but it's a poor substitute for being able to reason about what the performance of your queries will be.
> Screw no-SQL. They are just Object Oriented DBs (which were a disaster in the 90s) in new clothes.
On the contrary - the problems with SQL databases are much the same as the problems with object databases.
Not inserting bad data is much better than trying to make sense of inconsistent data because you don't know what is true or not.
What. Who's letting people delete random shit out of the DB? Why is the origin of this data not validating against the actual data from the DB before sending data?
"Hey we have to ensure that customers are selecting valid cities.
We have a list of cities in the DB. Let's ensure we cache the cities in the api server so that we're not hitting the DB for every page load for data that will very rarely, if ever, change.
Especially since we revoked Dave's DELETE permissions after that last time he deleted stuff from a production database."
It's on the client side, so it can't hold an open transaction with the server. It can validate against a previous version of the list of cities, but there's always the possibility of a delete happening in parallel.
(And if your answer is to use a delete flag rather than actually deleting a row, then you're undermining the referential integrity constraints that were supposed to be the whole point of using a relational database)
If a "city" is a) required attribute of an entity and b) required to exist as an entity itself, then deleting a city would require deleting all other entities that refer to that city.
If city is a nullable attribute, then your validation issue you raised earlier is voided - an invalid "city" can just be replaced with null.
Assuming you're wanting to delete cities, then your client side shouldn't be firing off data and assuming success.
> It can validate against a previous version of the list of cities, but there's always the possibility of a delete happening in parallel.
Yep, caching is hard, but your argument that "RDBMS aren't that great" is predicated on a very... ...convoluted system.
Right, so usually what people implement (and what the database enforces by default) is that you can't delete a city if there are any customers in that city. But they don't think about what happens when two people try to make changes in parallel, because the whole point of ACID is that each query runs in its own transaction.
> Yep, caching is hard, but your argument that "RDBMS aren't that great" is predicated on a very... ...convoluted system.
All the use cases for referential integrity - and transactionality - are like this though. You can't actually use them for anything, not with a web app where the client can't hold a transaction open.
You do know that the D in ACID stands for consistency? That is in case a transaction try to modify something that was already changed in a parallel transaction that was already committed, it will fail and will be rolled back (given proper transaction levels).
So I really don’t understand your point. Also, you can hold a transaction open for a client..
It will fail and be rolled back and you will lose the user's data that was written in it, unless you effectively build your own datastore on top of what the database offers.
> Also, you can hold a transaction open for a client..
For a client that's potentially gone offline for an indefinite period? Certainly not. Even if you're heartbeating that the client still has the form open somehow, that seems like a very suspect architecture.
The point of relational integrity is ensuring all your data is in a valid state. If you don't want to lose a client's data because you deleted a city, there are multiple engineering approaches open to you.
1) Don't delete cities, just don't. The semantics of doing so don't make any sense in the first instance.
2) Further, deleting entities from an RDBMS is pretty damn final. What if a business analyst wanted to do some historical analysis? Well, that city is gone now, so it won't be there. This is where "soft delete" comes in (And no, soft delete doesn't destroy referential integrity) - you've marked the entity as no longer valid for use, but you still retain your data surrounding it.
3) Don't respond with a 2xx until the data store acknowledges successful insertion.
4) Use Debezium or similar to broadcast entity changes to a Kafka topic keyed on entity id. If you want to retain all changes (e.g., for event streaming), then use an appropriate retention strategy. If you want to retain only the latest state, use log compaction - it will retain the latest record with a given key.
This makes it easy for an app being spun up to obtain the entity state without touching the DB, and by continuing to subscribe to the topic, update its cache to match the source of truth.
Of course, you don't have to use Kafka, it's just one approach, but many other systems will allow you to do the same.
Basically, your objections say nothing about the usefulness of a relational datastore, but rather, the need for careful engineering when building overly distributed systems.
And through all this, I'm wondering how you imagine a non-relational datastore solving these problems any better.
Right, and doing that at the lowest level of your datastore is a fundamentally misguided approach, because it means you can't ever store data that's in an invalid state (almost by definition). So when data does become invalid, you're effectively forced to destroy it.
> And no, soft delete doesn't destroy referential integrity
Yes it does. It destroys the property you described above - that you ensure all your data is in a valid state.
> Don't respond with a 2xx until the data store acknowledges successful insertion.
So the user sees an error. That doesn't actually help much, because what can they do with that error?
> Use Debezium or similar to broadcast entity changes to a Kafka topic keyed on entity id. If you want to retain all changes (e.g., for event streaming), then use an appropriate retention strategy. If you want to retain only the latest state, use log compaction - it will retain the latest record with a given key.
Yes, now keep tugging on the thread of that thought. If you try to write code to reconstruct the state of your relational database based on those changes, you've got two copies of your logic that will inevitably get out of sync, and you'll have bugs that you only discover when you try to actually do it. (And if you only record differences between writes that actually made it into the relational database, you haven't solved the original problem of data being lost because writes are rejected). What you want to do is instead make those Kafka events the primary "source of truth" and construct the "current state of the world" based on that, i.e. event sourcing.
You don't, and can't, make use of transactions with that approach - you get (eventual) consistency because each log is ordered, which gives you the properties that you want with less of the downsides (deadlocks), but writing and reading the eventual results of that write is a fundamentally async process (which is good in the long term - it forces you think about your dataflow and avoid loops - though it might involve more work upfront). And you don't really have relationality - if you need a relation in your live dataflow, you'll generally join at the event pipeline level and make the joined thing its own stream (much like a materialized view in SQL-land). You can build secondary indices etc. but those are explicitly a secondary thing layered on top of your primary datastore.
To be able to actually use database transactions you'd need them to extend across the form's lifetime. You can do that with a local application, but not with a webapp with the web request cycle.
> What about storing the results in memory until you can make sure it is successfully stored?
That has all the same problems as storing your data in memory and not using a datastore. In this scenario the data can never be stored because it now violates referential integrity.
> Also, what language/framework have you been using because these things are routinely handled by any that worth anything.
Many of the big-name Python or JVM-based ones, but it's a fundamental problem that can't be solved at that level. Which language/framework are you claiming handles this well?
You should probably tell the millions of companies actually using them for things that they're wrong. I'm sure they'll feel very silly.
Every data task becomes a programming problem which needs a team to work on to get something that might be stale. With SQL there's a lot of work. But you have a DB that everyone knows. 3rd party tools work. Developer on-boarding is easy. Your boss asks you how many users used feature X in the past month you can actually tell him right away...
Performance is actually good and consistent. Your data is also consistent and respects acid principles. The whole write performance of NoSQL DBs is a bit of a crock that doesn't stand the test of scaling. With proper cashing SQL is as performant when done right and the profiling tools are better. Screw no-SQL. They are just Object Oriented DBs (which were a disaster in the 90s) in new clothes.