Why SQL is beating NoSQL, and what this means for the future of data

Re:Why SQL is beating NoSQL, and what this means for the fut... by jjirsa on Tuesday September 26, 2017 @12:34AM (#15336199)

Many people ARE using petabyte scale databases.

They don't start out that way, but they will be if they're successful, and if they're not they probably shouldn't be running their own databases anyway (go RDS it up or something).

[ Parent | on HN ]

Re:Why SQL is beating NoSQL, and what this means for the fut... by happymellon on Tuesday September 26, 2017 @02:20AM (#15336616)

What? No they aren't, I work with satellite imagery processing which is quite large in it raw data format, and after a decade we are not dealing with petabytes of active data, hundreds of gigs for a full earth coverage.

Before that I have held positions in finance, dealing with realtime transaction processing. We did not work in petabytes.

If you are working in petabytes you are storing crap in your production database, and 99% of that data is wasted.

[ Parent | on HN ]

Re:Why SQL is beating NoSQL, and what this means for the fut... by altstar on Wednesday September 27, 2017 @07:05AM (#15346563)

Yeah right. We are using nosql in manufacturing. One second data for thousands of instruments create a lot of data very quickly. And that is per plant. Having a lot of plants at various sites in the world adds up.

[ Parent | on HN ]

Re:Why SQL is beating NoSQL, and what this means for the fut... by jjirsa on Wednesday September 27, 2017 @07:41AM (#15346771)

Exactly this. Whenever you collect millions of things, even at low resolution, it adds up quickly

Collecting high resolution and filtering down to low resolution for lower importance is still difficult and can require petabytes for real world use cases

[ Parent | on HN ]

Re:Why SQL is beating NoSQL, and what this means for the fut... by GrayShade on Tuesday September 26, 2017 @02:55AM (#15336720)

> What? No they aren't, I work with satellite imagery processing which is quite large in it raw data format, and after a decade we are not dealing with petabytes of active data, hundreds of gigs for a full earth coverage.

This get large rather fast when processing time series from multiple sensors.

But I agree with you otherwise, and it's not like you're storing satellite products in a database.

Now I'm curious what NSA is using :).

[ Parent | on HN ]

Re:Why SQL is beating NoSQL, and what this means for the fut... by ddorian43 on Tuesday September 26, 2017 @06:10AM (#15337445)

https://accumulo.apache.org/

See (History): https://en.wikipedia.org/wiki/Apache_Accumulo

[ Parent | on HN ]

Re:Why SQL is beating NoSQL, and what this means for the fut... by GrayShade on Tuesday September 26, 2017 @06:20AM (#15337471)

Thanks, I wasn't aware of it.

[ Parent | on HN ]

Re:Why SQL is beating NoSQL, and what this means for the fut... by EvilGrin on Tuesday September 26, 2017 @10:16AM (#15338813)

You could just check their Github? :)

https://github.com/nationalsecurityagency

[ Parent | on HN ]

Re:Why SQL is beating NoSQL, and what this means for the fut... by eberkund on Tuesday September 26, 2017 @01:44AM (#15336479)

Unless the amount of data you are collecting is increasing at a vastly unsustainable rate, you will be making megabucks and be able to afford a team of engineers to undertake a major redesign for version 2. But picking technologies based on scale at the onset is going to make getting to that point all the more unlikely.

[ Parent | on HN ]