The relational model is, as I understand it, fully general, so this literally cannot be the case. There is a problem of “if you do not know the shape of the data in advance” and “if the RDBMSs available to you do not efficiently handle data of the particular shape and access pattern you are using”, but “relational” isn't an inherent property of data, it's a property of a data model, and any data that can be modelled can be modelled relationally. That doesn't mean it is efficiently handled by every, or any, existing RDBMS, though.
(1) Wide table with 100K analytical features keyed against a customer. Now if you need to combine a few dozen of these features for predictive purposes then how do you do that in SQL ? You can't. Since every SQL database has a tiny column limit. In Cassandra that is a trivial O(1) fetch.
(2) Time series data can be many orders of magnitude faster in JSON document stores where you can have embedded/nested records.
(3) Star schema with a fact table and thousands of dimensions. In a SQL database that is thousand joins. In a JSON document store those dimensions can be embedded as well as made available as relations and again that is an O(1) fetch.
There are plenty of data shapes and access patterns for which existing RDBMS implementations are inefficient, as I acknowledged. That's not a feature of whether the data is or is not relational—“relational” is a feature of a data model, and any data can be modelled relationally—but rather of the available RDBMS implementations.
> Wide table with 100K analytical features keyed against a customer. Now if you need to combine a few dozen of these features for predictive purposes then how do you do that in SQL ?
In terms of both the relational model and the SQL language, this is trivial. In most concrete SQL implementations, it is both inefficient and requires a concrete schema that diverges from the abstract relational model of the data, because you need multiple tables with a common key rather than a single table, even if abstractly it is a single relation in a normalized model. There's nothing fundamental about either SQL or the relational model that requires this, its just not the usage pattern common RDBMS storage implementations are optimized for.
> Time series data can be many orders of magnitude faster in JSON document stores where you can have embedded/nested records.
Yes, that's an efficiency of access patterns issue, not an “is the data relational” issue.
Of course, outside the strict relational model of data, SQL based object-relational databases like Oracle and Postgres have supported embedded and nested records (via the combination of composite and collection data types) since before “JSON object stores” existed.
And technically we can use Microsoft Excel as a photo editor. But that's just ridiculous. Just like it's ridiculous to say you can use relational models for any data structure even if the queries will never return.
I think a lot of this RDBMs can't do X, came from using a MySQL. It has improved some, but was garbage until recently.
I'm not sure how wide you're taking about, but PG support up to 1600 depending on the datatypes (there is a page size maximum). Oracle supports 1000 IIRC.
[1] https://docs.microsoft.com/en-us/sql/sql-server/maximum-capa...
There's a genuine need for good scalable options, and with stuff like google spanner, those don't necessarily need to be "eventually consistent" or nosql. The simpler datamodels were probably easier to create and solved the problem, now I think there'll be a rise of scalable and consistent databases, best of both worlds.
ORMs fill the gap of course and SQL w JSON types are really nice. So I agree with your last paragraph but the rise was due to speed of db and speed for the developer.
I cranked out a lot of apps under short deadlines w MongoDB way back in the day. And I moved to Elasticsearch and CouchDB awhile ago. I'll still use those where appropriate but now in MariaDB for most. The competition was good.
* Using the stored data outside the use cases imagined by the original developer is harder than it should be. (Oh you want a customer dashboard with this and this data correlated, ..)
* "Migrations" can be rather error-prone if developers don't pay attention. (It was stored as a string first, and now it's an integer, now how do we query that). Just because there's no database schema, doesn't mean there's no schema.
* It's hard to see what's actually in a json table/collection. You'd need to browse the code, see how it's used, and then browse through the available data.
Migrations are less of an issue if you use an object document mapper which is just a leaner ORM. You can enforce your "schema" in your code while being schema-less. But yes if the type changes it could alter your queryability or you might have to write a script to update all records to reflect the new type. That can be a pain of course.
Viewing JSON collections in NoSQL is nice compared to SQL. Most your data is already composed as a sensible document and if you have subcollections the clients usually allow you to fold them or expand and show all. That's a win over SQL IMO.
Regarding JSON columns in SQL - I tend to store config/settings there and it's better than what it used to be for that stuff (serialized data or base64 encoded). In Navicat JSON columns are viewable in single record mode but pretty print format would be ideal.
There have been scalable and strongly consistent databases since the invention of the concept of NoSQL i.e. HBase and Cassandra (CL=ALL).
And the idea that "learning and optimize SQL" would instantly change people's rationale for using these databases shows you don't understand them much at all. There are many factors that come into play. For example you can't use SQL databases for large scale feature engineering.
Is it something that makesme grab for SQL it is the need to do adhoc querying exploring data.
Care to elaborate?
* Schema-on-read: Makes it easier to ingest large amounts of data, and then do ad-hoc exploration. The schema is only determined when reading the data, which is a bit easier for one-off data exploration, you determine how to interpret the data when actually using it. Not appropriate for production systems though. For example, a customer gives you a few TBs of data, you dump it on hadoop, and query it with spark. It would slow you down if you first have to convert it to a relational schema. Again, only good for one-off stuff.
* Most SQL databases have column limits, so if you have a very large amount of features, I'd imagine you'd run into these limits.
* Scalability. Feature engineering is very parallelizable, most normal SQL databases (excluding stuff like cassandra) aren't trivial to scale.
Spanner is consistent, available and partition tolerant due to the extremely precise clock synchronization that google achieved on their hardware and network, an innovation I hope will spread. So yes, that's new.
I don't think I claimed sql was the only solution that ever made sense. I'm sure for some use cases, schema-on-read or no schema is appropriate. But most of those usecases are rather niche. NoSQL became popular for (web)application software, which in my opinion isn't an appropriate use case for most of these applications.
Edit: I misunderstood. Spanner is CP, but google claims here: https://storage.googleapis.com/pub-tools-public-publication-... that it's also practically CA, which is technically incorrect, and something I misunderstood before.
This is nonsensical; even in a universe without relativity and with instantaneous communication--an unrealistic world where being able to talk about synchronized clocks can even make sense (as otherwise you fundamentally must deal with an inability to decide the order of two events due to latency)--you still cause indeterminate simultaneity by introducing a partition: your perfectly accurate and infinitely precise timestamps on one side of the partition would no longer be comparable with timestamps on the other side of the partition for purposes of establishing causality between events, as the recent changes being made on one side of the partition "should have" taken place between any older changes on the other side and any transactions being performed right now, and yet they couldn't because they were unknown. Spammer either gives up availability (waiting to verify what happened) or consistency (accepting that it will have to fix the result later) when a partition occurs: they can't do both, fundamentally; the only extent to which there is an engineering challenge here is "maybe you can minimize the number of partitions you have", which I totally believe Google to have done; but, given a partition (which cannot be fully prevented), the tradeoff is clear and unavoidable.
https://storage.googleapis.com/pub-tools-public-publication-....
TLDR: It's technically CP, but google claims partitions are so rare people can assume it's CA as well (99.999% available).
I assumed TrueTime bypassed the CAP theorem, but apparently that's marketing bs. It's to ensure something called external consistency, which is important if you want to take consistent snapshots over a distributed system.
This sounds like an old-wives tail at this point.
Except it's not. I've seen it in person, over and over. I've gone into systems where people were complaining about the 'database is slow', but there were no indexes. I've seen systems pull back 100s of thousands of records and then sort on app server and take the first 50. I've seen what would have been a simple join with exists clause turned into many round trips to the db with loops and app code complexities where a query would have 100x easier to reason about.
It's not even about optimization yet, but taking the time to learn even the surface capabilities of the tool.
Although honestly, sql is more than 40 years old and can be really nasty. Stored procedures, each database has a different dialect, nullability comparisons, CTEs. It's very different from any other programming language. It's not that hard to imagine people finding it hard to learn, especially front-end web developers who'd rather not touch the backend. If people don't know how indexes work, then it'll perform really badly as well, which would make databases like mongodb all the more attractive.
Not all developers are formally schooled, a lot of people don't know what a relational database model is.
I've seen people:
* Iterate over entire tables to get to a small number of records
* Not using joins, having the ORM "magically" handle everything (lazily getting non-joined values in separate queries, resulting in thousands of queries)
* Denormalize everything, and then wonder why things are inconsistent and hard to query
From the list of bad practices you've "seen" people do in SQL, I would guess their comfort zone code is probably a hot mess as well.
I used to think this because I learned SQL right along with all my other coding. I've realized though it is a mindset shift to go from imperative to declarative, and to think mostly in set operations. That shift can be hard for otherwise good developers.
That may be true (though, when it comes to dealing with more complex reporting functions and per-database-implementation differences/quirks outside of the realm of ANSI SQL it may not be).
However, it misses the point. Most engineers that struggle with SQL struggle with it because it's hidden from them partially (they're composing queries from snippets/query builders that come from other code, and never get to see the schema directly since it's hidden behind migrators and management interfaces) or completely (ORMs). Because the actual queries being run on actual schemas are less obvious, people do the wrong thing a lot.
That's not to say that abstractions on top of SQL are always bad--perhaps they're over/mis-used, but having worked on massive codebases where every dev's interaction with the database was "write a query in text or with a select().where().from()-type thin wrapper" and codebases that were 100% Django ORM, I can say with confidence that neither approach scales well absent big investments in correctness and RDBMS education.
Because these technologies every day power the most demanding applications on the planet.
In that light, it might be best to exclude BigTable, since it's only available publically as a service.
Or maybe SQL isn't perfect for every use case and other developers should stop arrogantly belittling those that choose to architect their own systems a particular way.
They objectively can't be "drinking the koolaid" or cargoculting something, because they've specifically created their own solutions, to solve their own problems.
I specifically didn't say SQL is perfect - I said that they're very popular for the wrong reasons, and often are used in situations that a relational database would be a better solution.
Makes sense.
Specific databases solve specific problems. In my current company, for log storage and analysis, InfluxDB was our solution after testing both ES and postgress, but we did compare all the products, and we still do as we expand. In my previous company, the main database in the product was Zope. I agree that mongoDB may be better in some use case, but i strongly doubt that most company are running it because it's the most appropriate product.
To stretch the kool aid metaphor, they're the ones mixing the kool aid, not drinking it.
When your argument is based on personal attacks of the people using that thing, it sounds more like you have an emotional investment in the alternative rather than a rational argument.
Arguably the first "NoSQL" database was a DIT, generally accessed via the LDAP protocol. OpenLDAP is one of the better known open source instances, but Novell eDirectory was there in the early part of this century, as was MS AD.
In the case of Directory Services, it's a completely different approach to both SQL and the common "Document Store" approach of modern "NoSQL" options. Unless you go cowboy-mode, you're still adhering to schemas (albeit with the option to combine schemas), you still use indexed attributes to conduct searches, but you have a literal tree of objects with a more flexible layout (e.g. multi-value attributes without needing to use a JSON column) than with a regular SQL RDBMS.
The rise in popularity of 'modern' NoSQL options (most commonly document stores) is IMO driven by a combination of lazy developers and too much Kool-aid consumption.