News for hackers. Stuff that matters.
Re:The end of big data by datalopers on Thursday May 26, 2022 @11:54PM (#31525872)
Databricks/Spark is overpriced garbage for Python-toting data scientists/engineers/analysts who don't know how to model data or write SQL. Exactly like Hadoop was 15 years ago for the Java-toting data scientists/engineers.
Re:The end of big data by threeseed on Friday May 27, 2022 @04:24AM (#31527278)
Spark is open source and free.

Databricks is their hosted version just like Aurora is a hosted PostgreSQL.

The fact you don't seem to understand the difference makes me question if you are qualified to be making statements that it is garbage and we can simply write SQL. Because having worked on many large, big data projects SQL is often not the right tool for every use case.

Re:The end of big data by clpm4j on Friday May 27, 2022 @01:23AM (#31526250)
That might be hyperbolic, but directionally I agree. The interesting part is that there are enough of those people out there in enterprise who fall into that "don't really know what they're doing but want to try something cool" bucket to keep Databricks flush with cash for years on end.
Re:The end of big data by datalopers on Friday May 27, 2022 @02:10AM (#31526501)
Yep. MongoDB earned $873M in revenue over the past 12 months servicing tech debt created between ‘10 - ‘16. Databricks will be around a long time for the same reason. It’s a viable business strategy for sure.
Re:The end of big data by michael_j_ward on Friday May 27, 2022 @03:22AM (#31526915)
If one does not need petabytes of scale, but am otherwise interested in the data lineage / observability / workflow being sold by databricks, what would you suggest?

Some evaluation Criteria:

- Ease of maintenance and operation is almost paramount.

- It's fine if the solution never lives anywhere but 1 single virtual server that scales vertically (data might grow to a couple TB, but not PETA BYTES)

- Similarly, 20 9's is not a criteria. If the machine fails and it takes an hour till someone goes and re-deploys, that's fine.

- Declarative, reproducible deployment with an easy upgrade story would be great

- Ideal if the deployment can be run locally for quick developmnet

Re:The end of big data by nooorofe on Saturday May 28, 2022 @11:39AM (#31541163)
Re:The end of big data by lysecret on Friday May 27, 2022 @10:50AM (#31530307)
As a former "Python-toting data scientists/engineers/analysts who don't know how to model data or write SQL" I wholeheartedly agree.
Re:The end of big data by nojito on Friday May 27, 2022 @08:10AM (#31528591)
Spark is many orders of magnitude faster than "properly modeled" data
Re:The end of big data by alexott on Friday May 27, 2022 @03:21AM (#31526900)
Databricks includes pure SQL implementations - Databricks SQL, for example. Or for example, Delta Live Tables allow to write you pipelines using only SQL…