I work every day with a > 4TB database and its all on 1 machine (a very big machine) with some very fast SAN. If you have money, Oracle can get very big on the right hardware. I am willing to bet MySQL or Postgres could do very well on that sort of hardware too.
More like 5Tb these days.
At that point, even in RDBMS you have to stop with the joins and flatten. I have experienced these limits in Oracle and MSSQL around 20-30 million row tables.
Scalable fundamentals like database flattening, dimensional modeling etc are all what key value stores give you from the start. But really a good mix works best or project specific, just saying in the future with TB, PB of data the JOIN is a historic remnant.
It is possible one day that RDBMS will be seen as one of those evil optimizations that we made in our small relative worlds at the time.
It is possible one day that RDBMS will be seen as one of those evil optimizations that we made in our small relative worlds at the time.
I doubt that for the simple reason that relational databases are maths: the relational algebra and relational calculus. There's no such theoretical underpinning to object databases.
True, the relational architecture is moved to the object or code level, if you think about it in relation to size, it makes more sense there for the future.
The RDBMS was the brain but it is also the storage, code will now dictate how to use the storage and not the storage itself, the storage is becoming a component.
The proper scaling issue is, of course, with the cpu and memory of your single DB server - and its disk speed - which can only be expanded so much, and at exponentially increasing cost. It's conceivable that even with a fixed amount of data, say 100G, you might have to confront scaling issues with increasing site usage, long before you reached anywhere near the limits of your raw disk space.
Seeing the amount of people generalizing that to apply to all RDBMSs makes me think they don't have any DB-knowledge outside MySQL.
And fyi I prefer PostgreSQL. But MySQL isn't THAT bad, c'mon.
MySQL may be able to perform better, as I haven't done in depth tests I wouldn't know how far it can be stretched, but my point is that almost everyone complaining about the performance of the RDMS model seems to come from MySQL thinking that it represent the utmost limits of what relational databases can do, and that is quite sad.
MySQL is not that bad, no. With proper knowledge it can do a lot, and beyond its limits, capably configured, I am not sure any RDBMS can help. Facebook is not developing Cassandra because they can't afford Oracle.
For cases like that you don't have one source of data, you do almost like an AFTER INSERT trigger to update other flattened tables of data that you need.
For instance, where you would normally perform a JOIN you would just after save update a JOIN table or a VIEW that is created by incremental updates.
So then you could filter by many other things and really aren't limited too much by space to make those duplications.
It happens with large RDBMS systems anyways after about 5GB of data, same type of scalability decisions have to be made.
But yes the article is off and you are right, most likely systems will approach this with multiple solutions for specific problems.