News for hackers. Stuff that matters.
Re:Billions wasted on Hadoop startups, the same will eventua... by wongarsu on Friday May 31, 2019 @07:04AM (#20059812)
Relational databases work just fine for terabytes of data. Aparently there are postgres deployments with petabytes of data. There aren't all that many companies with datasets that require anything else. Google and Twitter have the problems Hadoop solves, but it's fair to say that approximately nobody else has those problems.
Re:Billions wasted on Hadoop startups, the same will eventua... by nimrody on Friday May 31, 2019 @08:41AM (#20060316)
Whether Postgres (or similar) can handle the data depends not only on the data size but also on what you want to do with the data.

I.e., if you are generating reports, running aggregations over a large amount of data you definitely need some parallelism and Postgres isn't designed to handle these loads (certainly not petabytes). Even aggregating 100's of GB probably requires (or at least is more cost effective using) multiple machines.

Now Hadoop may not be a particularly efficient solution unless you need 100's of machines. But there is a limit to what a non-parallel single machine database can do. There are other solutions in-between.

And you really don't have to be twitter or google to handle significant amount of data these days. People are recording much more data in the hope of generating new insights and do need tools to process that data.

Re:Billions wasted on Hadoop startups, the same will eventua... by anarazel on Friday May 31, 2019 @09:29AM (#20060648)
> I.e., if you are generating reports, running aggregations over a large amount of data you definitely need some parallelism and Postgres isn't designed to handle these loads (certainly not petabytes). Even aggregating 100's of GB probably requires (or at least is more cost effective using) multiple machines.

Aggregating 100s of GB isn't much of a problem for PG these days. Yes, you can be faster - obviously - but it works quite well. And the price for separate systems (duplicated infrastructure, duplicated data, out-of-sync systems, ...) is noticable as well.

But yea, for many petabytes of data you either have to go to an entirely different system, or use something like Citus.

Disclaimer: I work on PG, and I used to work for Citus. So I'm definitely biased.

Re:Billions wasted on Hadoop startups, the same will eventua... by throwaway082729 on Friday May 31, 2019 @10:23AM (#20061134)
I'm genuinely curious. Can PG handle hundreds of users querying 100s of GB, sometimes the same set of tables, at the same time?
Re:Billions wasted on Hadoop startups, the same will eventua... by anarazel on Friday May 31, 2019 @10:36AM (#20061266)
Well. You're going to run out of CPU and memory bandwidth pretty quickly. So you'd need replicas to share processing load.

But I honestly don't think hundreds of users each querying 100s of GBs is all that common.

Re:Billions wasted on Hadoop startups, the same will eventua... by throwaway082729 on Friday May 31, 2019 @12:16PM (#20062239)
Again, genuinely curious to learn if you've experience with enterprise companies using 'big data' technologies.
Re:Billions wasted on Hadoop startups, the same will eventua... by anarazel on Friday May 31, 2019 @02:38PM (#20063758)
Yes.