News for hackers. Stuff that matters.
Re:How NoSQL forced the evolution of a scalable relational d... by logophobia on Friday July 13, 2018 @03:06AM (#17521028)
I would say the the rise of nosql options is driven by the need for scalability by a few large companies that desperately need it, and cargo-culting by developers that don't actually need it, but want to be like google and don't want to take the time to properly learn and optimize sql.

There's a genuine need for good scalable options, and with stuff like google spanner, those don't necessarily need to be "eventually consistent" or nosql. The simpler datamodels were probably easier to create and solved the problem, now I think there'll be a rise of scalable and consistent databases, best of both worlds.

Re:How NoSQL forced the evolution of a scalable relational d... by exclusiv on Friday July 13, 2018 @03:22AM (#17521081)
It wasn't just devs that didn't know how to properly optimize SQL. Programming w a NoSQL db was a better experience because there was no object impedance mismatch.

ORMs fill the gap of course and SQL w JSON types are really nice. So I agree with your last paragraph but the rise was due to speed of db and speed for the developer.

I cranked out a lot of apps under short deadlines w MongoDB way back in the day. And I moved to Elasticsearch and CouchDB awhile ago. I'll still use those where appropriate but now in MariaDB for most. The competition was good.

Re:How NoSQL forced the evolution of a scalable relational d... by logophobia on Friday July 13, 2018 @04:27AM (#17521352)
I've seen some of those applications. It might be easier & faster to start with a schema-less json database, but I saw some serious maintenance issues with applications like that:

* Using the stored data outside the use cases imagined by the original developer is harder than it should be. (Oh you want a customer dashboard with this and this data correlated, ..)

* "Migrations" can be rather error-prone if developers don't pay attention. (It was stored as a string first, and now it's an integer, now how do we query that). Just because there's no database schema, doesn't mean there's no schema.

* It's hard to see what's actually in a json table/collection. You'd need to browse the code, see how it's used, and then browse through the available data.

Re:How NoSQL forced the evolution of a scalable relational d... by exclusiv on Friday July 13, 2018 @09:33PM (#17528254)
Some apps aren't subject to new use-cases and some aren't meant to live that long or have another developer either. And SQL isn't great for the use-cases that solutions like Elasticsearch solves.

Migrations are less of an issue if you use an object document mapper which is just a leaner ORM. You can enforce your "schema" in your code while being schema-less. But yes if the type changes it could alter your queryability or you might have to write a script to update all records to reflect the new type. That can be a pain of course.

Viewing JSON collections in NoSQL is nice compared to SQL. Most your data is already composed as a sensible document and if you have subcollections the clients usually allow you to fold them or expand and show all. That's a win over SQL IMO.

Regarding JSON columns in SQL - I tend to store config/settings there and it's better than what it used to be for that stuff (serialized data or base64 encoded). In Navicat JSON columns are viewable in single record mode but pretty print format would be ideal.

Re:How NoSQL forced the evolution of a scalable relational d... by threeseed on Friday July 13, 2018 @03:57AM (#17521218)
Not sure what you are talking about here.

There have been scalable and strongly consistent databases since the invention of the concept of NoSQL i.e. HBase and Cassandra (CL=ALL).

And the idea that "learning and optimize SQL" would instantly change people's rationale for using these databases shows you don't understand them much at all. There are many factors that come into play. For example you can't use SQL databases for large scale feature engineering.

Re:How NoSQL forced the evolution of a scalable relational d... by CuriousSkeptic on Friday July 13, 2018 @05:44AM (#17521624)
To me “large scale feature engineering” is precisely the kind of problem SQL excel at.

Is it something that makesme grab for SQL it is the need to do adhoc querying exploring data.

Care to elaborate?

Re:How NoSQL forced the evolution of a scalable relational d... by logophobia on Friday July 13, 2018 @07:38AM (#17522024)
He's probably refering to a combination of features that "nosql" databases have.

* Schema-on-read: Makes it easier to ingest large amounts of data, and then do ad-hoc exploration. The schema is only determined when reading the data, which is a bit easier for one-off data exploration, you determine how to interpret the data when actually using it. Not appropriate for production systems though. For example, a customer gives you a few TBs of data, you dump it on hadoop, and query it with spark. It would slow you down if you first have to convert it to a relational schema. Again, only good for one-off stuff.

* Most SQL databases have column limits, so if you have a very large amount of features, I'd imagine you'd run into these limits.

* Scalability. Feature engineering is very parallelizable, most normal SQL databases (excluding stuff like cassandra) aren't trivial to scale.

Re:How NoSQL forced the evolution of a scalable relational d... by logophobia on Friday July 13, 2018 @04:09AM (#17521265)
Cassandra is AP (available and partition tolerant), with tunable consistency. HBase is CP (consistent and partition tolerant, but not always available in case of network partitions).

Spanner is consistent, available and partition tolerant due to the extremely precise clock synchronization that google achieved on their hardware and network, an innovation I hope will spread. So yes, that's new.

I don't think I claimed sql was the only solution that ever made sense. I'm sure for some use cases, schema-on-read or no schema is appropriate. But most of those usecases are rather niche. NoSQL became popular for (web)application software, which in my opinion isn't an appropriate use case for most of these applications.

Edit: I misunderstood. Spanner is CP, but google claims here: https://storage.googleapis.com/pub-tools-public-publication-... that it's also practically CA, which is technically incorrect, and something I misunderstood before.

Re:How NoSQL forced the evolution of a scalable relational d... by saurik on Friday July 13, 2018 @05:33AM (#17521575)
> Spanner is consistent, available and partition tolerant due to the extremely precise clock synchronization...

This is nonsensical; even in a universe without relativity and with instantaneous communication--an unrealistic world where being able to talk about synchronized clocks can even make sense (as otherwise you fundamentally must deal with an inability to decide the order of two events due to latency)--you still cause indeterminate simultaneity by introducing a partition: your perfectly accurate and infinitely precise timestamps on one side of the partition would no longer be comparable with timestamps on the other side of the partition for purposes of establishing causality between events, as the recent changes being made on one side of the partition "should have" taken place between any older changes on the other side and any transactions being performed right now, and yet they couldn't because they were unknown. Spammer either gives up availability (waiting to verify what happened) or consistency (accepting that it will have to fix the result later) when a partition occurs: they can't do both, fundamentally; the only extent to which there is an engineering challenge here is "maybe you can minimize the number of partitions you have", which I totally believe Google to have done; but, given a partition (which cannot be fully prevented), the tradeoff is clear and unavoidable.

Re:How NoSQL forced the evolution of a scalable relational d... by qaq on Friday July 13, 2018 @05:29AM (#17521564)
Spanner is CP I think the A part comes from reliability of google network infrastructure which is not part of DB system's design per se.
Re:How NoSQL forced the evolution of a scalable relational d... by logophobia on Friday July 13, 2018 @05:39AM (#17521608)
I think I misunderstood that yes.

https://storage.googleapis.com/pub-tools-public-publication-....

TLDR: It's technically CP, but google claims partitions are so rare people can assume it's CA as well (99.999% available).

I assumed TrueTime bypassed the CAP theorem, but apparently that's marketing bs. It's to ensure something called external consistency, which is important if you want to take consistent snapshots over a distributed system.

Re:How NoSQL forced the evolution of a scalable relational d... by majewsky on Friday July 13, 2018 @09:00AM (#17522536)
Semi-related soapbox: Can we all please stop saying "CAP theorem" and instead say "CAP rule"? The formulation of this supposed theorem is so vague that it can neither be proven nor disproven.
Re:How NoSQL forced the evolution of a scalable relational d... by logophobia on Friday July 13, 2018 @11:22AM (#17523674)
It was proven in 2002, has a formal definition, so I'm going with calling it a theorem. Maybe this'll help: https://mwhittaker.github.io/blog/an_illustrated_proof_of_th... ?
Re:How NoSQL forced the evolution of a scalable relational d... by 189723954 on Friday July 13, 2018 @04:21AM (#17521319)
>don't want to take the time to properly learn and optimize sql.

This sounds like an old-wives tail at this point.

Re:How NoSQL forced the evolution of a scalable relational d... by matwood on Friday July 13, 2018 @07:08AM (#17521908)
> This sounds like an old-wives tail at this point.

Except it's not. I've seen it in person, over and over. I've gone into systems where people were complaining about the 'database is slow', but there were no indexes. I've seen systems pull back 100s of thousands of records and then sort on app server and take the first 50. I've seen what would have been a simple join with exists clause turned into many round trips to the db with loops and app code complexities where a query would have 100x easier to reason about.

It's not even about optimization yet, but taking the time to learn even the surface capabilities of the tool.

Re:How NoSQL forced the evolution of a scalable relational d... by logophobia on Friday July 13, 2018 @04:42AM (#17521406)
I've met developers like that.

Although honestly, sql is more than 40 years old and can be really nasty. Stored procedures, each database has a different dialect, nullability comparisons, CTEs. It's very different from any other programming language. It's not that hard to imagine people finding it hard to learn, especially front-end web developers who'd rather not touch the backend. If people don't know how indexes work, then it'll perform really badly as well, which would make databases like mongodb all the more attractive.

Not all developers are formally schooled, a lot of people don't know what a relational database model is.

I've seen people:

* Iterate over entire tables to get to a small number of records

* Not using joins, having the ORM "magically" handle everything (lazily getting non-joined values in separate queries, resulting in thousands of queries)

* Denormalize everything, and then wonder why things are inconsistent and hard to query

Re:How NoSQL forced the evolution of a scalable relational d... by sqlcook on Friday July 13, 2018 @04:59AM (#17521467)
It actually IS quite hard to imagine solid engineers finding it hard to learn simple SQL. It's a lot harder to write "nasty" sql than "nasty" js, and if you're a front-end dev struggling with sql, then you probably should not be doing back end work. Yes not all databases are equal, thats why there are ORMs and ANSI standards.

From the list of bad practices you've "seen" people do in SQL, I would guess their comfort zone code is probably a hot mess as well.

Re:How NoSQL forced the evolution of a scalable relational d... by matwood on Friday July 13, 2018 @07:12AM (#17521923)
> It actually IS quite hard to imagine solid engineers finding it hard to learn simple SQL.

I used to think this because I learned SQL right along with all my other coding. I've realized though it is a mindset shift to go from imperative to declarative, and to think mostly in set operations. That shift can be hard for otherwise good developers.

Re:How NoSQL forced the evolution of a scalable relational d... by zbentley on Friday July 13, 2018 @08:34AM (#17522358)
> It's a lot harder to write "nasty" sql than "nasty" js

That may be true (though, when it comes to dealing with more complex reporting functions and per-database-implementation differences/quirks outside of the realm of ANSI SQL it may not be).

However, it misses the point. Most engineers that struggle with SQL struggle with it because it's hidden from them partially (they're composing queries from snippets/query builders that come from other code, and never get to see the schema directly since it's hidden behind migrators and management interfaces) or completely (ORMs). Because the actual queries being run on actual schemas are less obvious, people do the wrong thing a lot.

That's not to say that abstractions on top of SQL are always bad--perhaps they're over/mis-used, but having worked on massive codebases where every dev's interaction with the database was "write a query in text or with a select().where().from()-type thin wrapper" and codebases that were 100% Django ORM, I can say with confidence that neither approach scales well absent big investments in correctness and RDBMS education.

Re:How NoSQL forced the evolution of a scalable relational d... by jasonlotito on Friday July 13, 2018 @10:05AM (#17523066)
Just read through these threads. There are people literally saying these things.