News for hackers. Stuff that matters.

A comment on Databricks response to Snowflake's accusation of lacking integrity. Parent comment

Re:Databricks response to Snowflake's accusation of lacking ... by hello_moto on Monday November 15, 2021 @07:50PM (#29235127)
It'll take a few more years until these companies fixed all the bugs and address all the scalability issues.

As of today, these companies are not good enough to take on the Data Warehouse part.

[ Parent | on HN ]
Re:Databricks response to Snowflake's accusation of lacking ... by feqgmmr2 on Monday November 15, 2021 @09:09PM (#29235842)
Spark has always been able to handle way larger scale than any DW.
[ Parent | on HN ]
Re:Databricks response to Snowflake's accusation of lacking ... by hello_moto on Monday November 22, 2021 @01:02AM (#29303308)
Handle what though?

Can Spark queries 100Bn structured data performing aggregation on multiple fields (or dimension?)

[ Parent | on HN ]
Re:Databricks response to Snowflake's accusation of lacking ... by geoduck14 on Monday November 15, 2021 @09:49PM (#29236153)
In my previous company, we had 63 petabytes of data in Snowflake.
[ Parent | on HN ]
Re:Databricks response to Snowflake's accusation of lacking ... by hello_moto on Monday November 22, 2021 @01:03AM (#29303316)
That sounds great: storage problem is solved.

What about large scale read via OLAP queries (y'know, the typical measures and dimensions)

[ Parent | on HN ]
Re:Databricks response to Snowflake's accusation of lacking ... by blueglassfish on Tuesday November 16, 2021 @02:42AM (#29237933)
That's a respectable amount for a DW, true. Spark and it's ilk are designed for much larger scales though. Multiple FAANG use cases for Spark are in the petabytes per week range.
[ Parent | on HN ]