News for hackers. Stuff that matters.

A comment on Databricks Strikes $1.3B Deal for Generative AI Startup MosaicML. Parent comment

Re:Databricks Strikes $1.3B Deal for Generative AI Startup M... by aynyc on Monday June 26, 2023 @11:25AM (#36480920)
Their moat was to be able to spin up spark clusters for you. This was very useful when companies just starting in that space and don’t have the skills. But with K8s and clusters becoming more common for normal workload, it’s not hard to manage your own spark clusters anymore. There goes their moat. Their other offerings just aren’t that amazing. People now don’t just want to buy hand tools anymore, they want power tools. AI/ML and its cousins are in the power tool space.

Databrick cost is stupid high, like snowflake. Any companies looking to chop budget would put those contracts up first.

[ Parent | on HN ]
Re:Databricks Strikes $1.3B Deal for Generative AI Startup M... by disgruntledphd2 on Monday June 26, 2023 @11:57AM (#36481469)
Their moat is their phenomenal sales team.

Like, I'm at company 3 that pays for databricks.

Amount of those companies that use Spark = 0.

I've given up complaining now.

[ Parent | on HN ]
Re:Databricks Strikes $1.3B Deal for Generative AI Startup M... by esafak on Monday June 26, 2023 @01:14PM (#36482686)
So what are they using, the notebooks?
[ Parent | on HN ]
Re:Databricks Strikes $1.3B Deal for Generative AI Startup M... by pletnes on Monday June 26, 2023 @05:34PM (#36486193)
But in the notebooks you run pyspark / scala spark / spark sql code?
[ Parent | on HN ]
Re:Databricks Strikes $1.3B Deal for Generative AI Startup M... by esafak on Monday June 26, 2023 @11:02PM (#36488402)
Yes, you can, if you want.
[ Parent | on HN ]
Re:Databricks Strikes $1.3B Deal for Generative AI Startup M... by disgruntledphd2 on Tuesday June 27, 2023 @07:09AM (#36491304)
Yes, but for most of these companies they don't actually run (much or any) spark stuff, they just use the notebooks.

It's really depressing, tbh but I've made my peace with it now.

[ Parent | on HN ]
Re:Databricks Strikes $1.3B Deal for Generative AI Startup M... by pletnes on Tuesday June 27, 2023 @12:00PM (#36494697)
I can’t blame them really. Regular pandas-in-notebook can deal with quite a bit of data and is reasonably convenient. Even if the org and datasets are large, it’s rare to have huge single tables. Then you can go full spark when you have to.
[ Parent | on HN ]
Re:Databricks Strikes $1.3B Deal for Generative AI Startup M... by disgruntledphd2 on Wednesday June 28, 2023 @03:52AM (#36503592)
Sure, but why bother paying databricks for that? By default you'll be running 3+ machines while doing all your work in memory on the master node. It's just so very, very silly.
[ Parent | on HN ]
Re:Databricks Strikes $1.3B Deal for Generative AI Startup M... by pletnes on Wednesday June 28, 2023 @09:57AM (#36506564)
True. I’ve been wondering why they don’t facilitate non-spark use better. But then you can’t reasonably invoice 3 VMs extra, obviously.
[ Parent | on HN ]
Re:Databricks Strikes $1.3B Deal for Generative AI Startup M... by pletnes on Monday June 26, 2023 @05:36PM (#36486211)
So Databricks is being replaced by open source, self-managed spark-on-k8s? Broadly speaking? Any concrete recommendations on how to do that in a project?
[ Parent | on HN ]
Re:Databricks Strikes $1.3B Deal for Generative AI Startup M... by splendor_spoon on Monday June 26, 2023 @07:28PM (#36487203)
We tried this at a previous company I was at, it was incredibly tough to maintain and tune. We ended up using databricks which made it significantly more productive.
[ Parent | on HN ]
Re:Databricks Strikes $1.3B Deal for Generative AI Startup M... by aynyc on Monday June 26, 2023 @07:50PM (#36487376)
No, it's not about the technology. It's about your stack and skill set. If you are still on the school of one box for all. Then just pay for Databrick or EMR. If you are already running K8s because you invested into that world, then adding Spark isn't a huge jump.
[ Parent | on HN ]