Why SQL is beating NoSQL, and what this means for the future of data

Re:Why SQL is beating NoSQL, and what this means for the fut... by jjirsa on Tuesday September 26, 2017 @12:03AM (#15336041)

The reason people don’t start with a relational DB has nothing to do with the data model or query language - people choose databases like Cassandra because scaling to 500 Postgres or MySQL instances holding a combined petabyte of data is horrific, but it’s dirt simple in things like Cassandra

The query language is a side effect of the underlying storage engine - you don’t choose it because you want a key value store, you choose it because you want horizontal scalability and cross wan HA

[ Parent | on HN ]

Re:Why SQL is beating NoSQL, and what this means for the fut... by 6nf on Tuesday September 26, 2017 @12:28AM (#15336170)

You make it sound like many people are using petabyte scale databases. This is not even close to true. ESPECIALLY when you're just starting out. Most new projects will start with almost no data at all and the vast majority of projects will never ever get to the petabyte scales.

[ Parent | on HN ]

Re:Why SQL is beating NoSQL, and what this means for the fut... by jjirsa on Tuesday September 26, 2017 @12:34AM (#15336199)

Many people ARE using petabyte scale databases.

They don't start out that way, but they will be if they're successful, and if they're not they probably shouldn't be running their own databases anyway (go RDS it up or something).

[ Parent | on HN ]

Re:Why SQL is beating NoSQL, and what this means for the fut... by happymellon on Tuesday September 26, 2017 @02:20AM (#15336616)

What? No they aren't, I work with satellite imagery processing which is quite large in it raw data format, and after a decade we are not dealing with petabytes of active data, hundreds of gigs for a full earth coverage.

Before that I have held positions in finance, dealing with realtime transaction processing. We did not work in petabytes.

If you are working in petabytes you are storing crap in your production database, and 99% of that data is wasted.

[ Parent | on HN ]

Re:Why SQL is beating NoSQL, and what this means for the fut... by altstar on Wednesday September 27, 2017 @07:05AM (#15346563)

Yeah right. We are using nosql in manufacturing. One second data for thousands of instruments create a lot of data very quickly. And that is per plant. Having a lot of plants at various sites in the world adds up.

[ Parent | on HN ]

Re:Why SQL is beating NoSQL, and what this means for the fut... by jjirsa on Wednesday September 27, 2017 @07:41AM (#15346771)

Exactly this. Whenever you collect millions of things, even at low resolution, it adds up quickly

Collecting high resolution and filtering down to low resolution for lower importance is still difficult and can require petabytes for real world use cases

[ Parent | on HN ]

Re:Why SQL is beating NoSQL, and what this means for the fut... by GrayShade on Tuesday September 26, 2017 @02:55AM (#15336720)

> What? No they aren't, I work with satellite imagery processing which is quite large in it raw data format, and after a decade we are not dealing with petabytes of active data, hundreds of gigs for a full earth coverage.

This get large rather fast when processing time series from multiple sensors.

But I agree with you otherwise, and it's not like you're storing satellite products in a database.

Now I'm curious what NSA is using :).

[ Parent | on HN ]

Re:Why SQL is beating NoSQL, and what this means for the fut... by ddorian43 on Tuesday September 26, 2017 @06:10AM (#15337445)

https://accumulo.apache.org/

See (History): https://en.wikipedia.org/wiki/Apache_Accumulo

[ Parent | on HN ]

Re:Why SQL is beating NoSQL, and what this means for the fut... by GrayShade on Tuesday September 26, 2017 @06:20AM (#15337471)

Thanks, I wasn't aware of it.

[ Parent | on HN ]

Re:Why SQL is beating NoSQL, and what this means for the fut... by EvilGrin on Tuesday September 26, 2017 @10:16AM (#15338813)

You could just check their Github? :)

https://github.com/nationalsecurityagency

[ Parent | on HN ]

Re:Why SQL is beating NoSQL, and what this means for the fut... by eberkund on Tuesday September 26, 2017 @01:44AM (#15336479)

Unless the amount of data you are collecting is increasing at a vastly unsustainable rate, you will be making megabucks and be able to afford a team of engineers to undertake a major redesign for version 2. But picking technologies based on scale at the onset is going to make getting to that point all the more unlikely.

[ Parent | on HN ]

Re:Why SQL is beating NoSQL, and what this means for the fut... by bildung on Tuesday September 26, 2017 @09:32AM (#15338448)

Postgres (as an example) has no upper limit for database size, and the upper limit for a single table was 64TB a while ago, perhaps higher now. And you can buy stock hardware with 6 TB RAM and 76TB SSD e.g. from Dell.

If you max out that single server, you could easily hire one database engineer to care about horizontal scaling.

[ Parent | on HN ]

Re:Why SQL is beating NoSQL, and what this means for the fut... by jbergens on Wednesday September 27, 2017 @02:18AM (#15345315)

It doesn't help when the nbr of queries and updates increases a lot. For many systems having TB of data also means having thousands or millions of users.

You also probably want backups and possibly slaves that are updated inside transactions to avoid data loss. These things can lower your throughput even more.

[ Parent | on HN ]

Re:Why SQL is beating NoSQL, and what this means for the fut... by jjirsa on Tuesday September 26, 2017 @05:15PM (#15342582)

Show me a single person who runs 64TB on a single table on a single node, and I'll show you someone who will cry because of a prod failure sooner than later.

[ Parent | on HN ]

Re:Why SQL is beating NoSQL, and what this means for the fut... by collyw on Tuesday September 26, 2017 @03:18PM (#15341695)

The vast majority of hipster NoSQL developers I have debated with are using a single machine.

[ Parent | on HN ]

Re:Why SQL is beating NoSQL, and what this means for the fut... by jjirsa on Tuesday September 26, 2017 @05:14PM (#15342574)

Those people are objectively wrong.

[ Parent | on HN ]

Re:Why SQL is beating NoSQL, and what this means for the fut... by collyw on Wednesday September 27, 2017 @03:55PM (#15351678)

You try and explain that to them, but whatever database they are using "is fast". Not that I have ever had a problem with relational databases being slow. Certain queries, yes, but a bit of rewriting them and proper indexing usually solves that.

[ Parent | on HN ]