Databricks and Neon

Re:Databricks and Neon by kwillets on Wednesday May 14, 2025 @11:53AM (#43986007)

It's been a commodity for decades now. Metrics like price-performance have a long history, but the SnowBricks products fail at them quite dramatically. The difference is hard-sell vs. soft or no-sell.

[ Parent | on HN ]

Re:Databricks and Neon by datadrivenangel on Wednesday May 14, 2025 @01:34PM (#43987166)

Not having to buy an appliance and pay for it up front is quite a valuable option. Also the split between processing and storage allows for better archival and scaling strategies.

[ Parent | on HN ]

Re:Databricks and Neon by hlpn on Wednesday May 14, 2025 @10:29AM (#43984990)

Totally agree. Happy open source StarRocks user here using the k8s operator for customer-facing analytics on terabytes of data. There's very little need for Databricks in my world.

[ Parent | on HN ]

Re:Databricks and Neon by anilshanbhag on Wednesday May 14, 2025 @11:11AM (#43985507)

Looking at StarRocks site (https://www.starrocks.io/), they compare against Clickhouse, Druid and Trino. Don't even compare against Spark/Databricks! Guess Spark is just not competitive.

[ Parent | on HN ]

Re:Databricks and Neon by jflkdsjlcsuq on Thursday May 15, 2025 @12:29AM (#43991854)

Don't fall for benchmarketing. Do your own benchmarks for your use case.

[ Parent | on HN ]

Re:Databricks and Neon by lars_francke on Wednesday May 14, 2025 @11:11AM (#43985511)

Anyone looking for an open-source Cloudera alternative based on Kubernetes operators. We're building one (~5 years old now): https://stackable.tech/ & https://github.com/stackabletech/

On-premise open-source S3 is a problem though. MinIO is not something we're touching and other than that it looks a bit empty with enterprise ready solutions.

[ Parent | on HN ]

Re:Databricks and Neon by matt-p on Wednesday May 14, 2025 @11:51AM (#43985979)

What's wrong with minio out of curiosity? Ceph an option?

[ Parent | on HN ]

Re:Databricks and Neon by lars_francke on Wednesday May 14, 2025 @12:08PM (#43986190)

This is at least partially subjective.

https://news.ycombinator.com/item?id=32148007

https://news.ycombinator.com/item?id=35299665

Ceph would be a theoretical option, but a) we don't have a lot of experience with it and b) it's relatively complex to operate. We'd really love to add a lighter option to our stack that's under the stewardship of a foundation.

[ Parent | on HN ]

Re:Databricks and Neon by positisop on Wednesday May 14, 2025 @01:48PM (#43987329)

Try expanding a cluster, or changing erasure coding configuration, or using anything that needs random access within a file (parquet), or any day 2 operation.

[ Parent | on HN ]

Re:Databricks and Neon by matt-p on Wednesday May 14, 2025 @01:54PM (#43987382)

on what?

[ Parent | on HN ]

Re:Databricks and Neon by positisop on Wednesday May 14, 2025 @02:16PM (#43987596)

Look under the hood, the limitations are based on the core, sticking a UI on it does not hide what needs to happen at scale.

[ Parent | on HN ]

Re:Databricks and Neon by dilyevsky on Wednesday May 14, 2025 @02:50PM (#43987958)

Guessing you’re referring to minio not ceph? Have they still not figured out how to do day 2? I mainly avoid them because of their license and the way they interpret it

[ Parent | on HN ]

Re:Databricks and Neon by positisop on Wednesday May 14, 2025 @03:49PM (#43988497)

They are not efficient; they have a one-time static hash to create a cluster. After that, it is all duct tape and glue. Want to expand? Add another cluster (pool) and then look for the cluster that contains the object. They don't know which cluster has the object, and performance does not scale as well with additional clusters. Want to decommission a single node, drain the cluster. They refer to multiple pools as a single cluster, but it is essentially a set of static hashes that lack the intelligence to locate objects. Got the initial EC configuration not quite right.. sorry need to redo the entire cluster.

MinIO is a good fit if you want a small cluster that doesn't require day 2 operational complexity, as you only store a few TBs.

I have not looked into them recently, but I doubt the core has changed. Being VC-funded and looking for an exit makes them overinvest in marketing and story telling.

[ Parent | on HN ]

Re:Databricks and Neon by dilyevsky on Wednesday May 14, 2025 @03:51PM (#43988517)

That tracks with my past analysis as well, thanks

[ Parent | on HN ]

Re:Databricks and Neon by cirque on Wednesday May 14, 2025 @06:57PM (#43990049)

Even some basic s3 storage patterns weren’t considered when the core storage scheme was designed. Lacks an index and depends on filesystem to organize objects and then crumbles to lock contention when too many versions are stored or under walkdir calls when anything is listed. It also can’t even support writing to the same set of keys as S3 should allow since it implicitly depends on underlying filesystem paths.

They might have added an index by now but gatekept it to their enterprise AIStor offering since they’ve abandoned any investment in open source at this point or appearance that they care about that. Their initial inclination in response to this issue says everything - https://github.com/minio/minio/issues/20845#issuecomment-259...

[ Parent | on HN ]

Re:Databricks and Neon by bittermandel on Wednesday May 14, 2025 @01:07PM (#43986862)

Wouldn't Rook be a good solution? It's definitely proven in much larger settings than Minio, as it's just Ceph.

[ Parent | on HN ]

Re:Databricks and Neon by SOLAR_FIELDS on Wednesday May 14, 2025 @03:05PM (#43988120)

Don’t SeaweedFS and ceph/rook also offer this? Ceph/rook is definitely enterprise ready

[ Parent | on HN ]

Re:Databricks and Neon by pjdbruin on Wednesday May 14, 2025 @04:12PM (#43988688)

Great to see cost-effective alternatives to Cloudera and Databricks! We’ve spent three years building IOMETE, a self-hosted data lakehouse that combines Apache Iceberg and Spark, designed to run natively on Kubernetes. We’re focused on on-premises deployments to address the growing need for data sovereignty and low TCO, with a streamlined setup for large-scale analytics. Early adopters are seeing strong results. Curious about your experience with Trino and Superset—any tips for optimizing performance at scale?

[ Parent | on HN ]

Re:Databricks and Neon by __turbobrew__ on Thursday May 15, 2025 @02:45AM (#43992460)

> On-premise open-source S3 is a problem though

Rook/ceph with object storage is pretty bulletproof: https://www.rook.io/docs/rook/v1.17/Storage-Configuration/Ob...

I do wish more systems had high quality operators out there. A lot of operators I have looked into are half baked, not reliable, or not supported.

[ Parent | on HN ]

Re:Databricks and Neon by datadrivenangel on Wednesday May 14, 2025 @01:32PM (#43987143)

But why would you buy an operational DB from Databricks? The only thing that makes sense is Databricks flailing to maintain market cap.

[ Parent | on HN ]

Re:Databricks and Neon by antruok on Wednesday May 14, 2025 @02:28PM (#43987703)

In addition to the AI use cases, sometimes you wanna share the data warehouse data in oltp way for fast lookups and high concurrency. Not sure whether Neon will do that but I hope so.

One example from Snowflake is hybrid tables which adds rowstore next to columnar.

OLAP + OLTP = HTAP

[ Parent | on HN ]

Re:Databricks and Neon by bbohan on Wednesday May 14, 2025 @03:55PM (#43988549)

SingleStore been doing that for years . Unistore been strugglin

[ Parent | on HN ]

Re:Databricks and Neon by antruok on Wednesday May 14, 2025 @04:12PM (#43988692)

I wonder why Singlestore has been so unpopular (at least I never hear about it). Quick guess is that HTAP itself isn’t a significant feature requirement, maybe just a cherry on top of other major db features.

[ Parent | on HN ]

Re:Databricks and Neon by orangechairs on Wednesday May 14, 2025 @04:26PM (#43988808)

SingleStore (nee MemSQL) was/is niche -- great fit inside banks in particular.

[ Parent | on HN ]

Re:Databricks and Neon by moonikakiss on Wednesday May 14, 2025 @04:35PM (#43988881)

it was a couple things. 1. It's really really hard to replace anyone's OLTP. 2. OLTP and OLAP are owned by such different teams. Who do you make your champion? 3. The modern HTAP dream is possible without something like SingleStore. You need a columnstore that can keep up with your OLTP tables and provide transactional correctness. Who cares if it's all within one system

ps: I worked at SingleStore. https://www.mooncake.dev/blog/htap-is-dead

[ Parent | on HN ]

Re:Databricks and Neon by ako on Wednesday May 14, 2025 @02:32PM (#43987750)

ETL to bring all your data into Databricks/Snowflake is a lot of effort. Much better if your OLTP data already exists in Databricks and you directly access it from your OLAP layer.

[ Parent | on HN ]

Re:Databricks and Neon by moonikakiss on Wednesday May 14, 2025 @03:56PM (#43988553)

With the push towards open table formats (Iceberg) from both Snowflake and Databricks, it's even harder to get your Postgres OLTP tables ready for OLAP.

The problem isn't in the CDC / replication tools in the market.

The problem is that columnar stores (especially Iceberg) are not designed for the write /upserts patterns of OLTP systems.

They just can't keep up...

This is a big problem we're hoping to solve at Mooncake [0]. Turn Iceberg into an operational columnstore. So that it can be keep up (<s freshness) with your Postgres.

https://www.mooncake.dev/

[ Parent | on HN ]

Re:Databricks and Neon by ako on Wednesday May 14, 2025 @05:02PM (#43989186)

Is Iceberg involved in every read/write? I thought it was mostly metadata?

[ Parent | on HN ]

Re:Databricks and Neon by zhousun on Thursday May 15, 2025 @01:27PM (#43997264)

DataFile(parquet) is not enough for table with update/delete, (they are part of iceberg "metadata"). for CDC from OLTP use-cases, the pattern involves rapidly marking rows as deleted/ insert new rows and optimizing small files. This is required for minutes-latency replication.

And for second latency replication, it is more involving, you actually need to build layer on top of iceberg to track pk/ apply deletion.

[ Parent | on HN ]

Re:Databricks and Neon by swyx on Wednesday May 14, 2025 @03:10PM (#43988160)

if Databricks just wanted a row DB they couldve done postgres themselves. paying this much for Neon i think is a sign that Neon has something special they want (which, knowing their marketing line, is "independently scalable storage and compute for postgres")

[ Parent | on HN ]

Re:Databricks and Neon by dustingetz on Wednesday May 14, 2025 @04:31PM (#43988843)

"Time is the denominator"

[ Parent | on HN ]

Re:Databricks and Neon by t0mas88 on Wednesday May 14, 2025 @06:01PM (#43989663)

That sounds like AWS Aurora?

[ Parent | on HN ]

Re:Databricks and Neon by swyx on Thursday May 15, 2025 @08:48PM (#44000795)

mashed w a lil bit of snowflake arch

[ Parent | on HN ]

Re:Databricks and Neon by yencabulator on Wednesday May 14, 2025 @06:42PM (#43989941)

Easy quick cheap forks of database state for AI agents to muck with.

[ Parent | on HN ]

Re:Databricks and Neon by richardw on Wednesday May 14, 2025 @03:37PM (#43988374)

Not commoditising for enterprise. My last gig wouldn’t allow open source software or any company that might not be there in a decade, or which kept data anywhere but our own tenant. We’d look for the “call us” pricing rather than hate it, which I normally do. We added databricks and it was considered one of my top three achievements, because they don’t have to think about data platforms again, just focus on using it. It’s SO expensive for an enterprise to rejig for a new platform that you can’t rely on (insert open source project here).

I managed to add one startup and so far it’s done very well, but it was an exceptional case and the global CEO wanted the functionality. But it used MongoDB and we didn’t have any skills, so rather then learn one tiny thing for an irrelevant data store they added cash to use Atlas with all the support and RBAC etc etc. Kept hiring load down, one number to call, job done.

Startups are from Venus, enterprise are from Jupiter.

[ Parent | on HN ]

Re:Databricks and Neon by antruok on Wednesday May 14, 2025 @04:07PM (#43988653)

Enterpise also often wants a full data platform (like Databricks), not a plain data warehouse.

[ Parent | on HN ]

Re:Databricks and Neon by CharlesW on Wednesday May 14, 2025 @10:35PM (#43991282)

Are there even any "plain data warehouse" vendors left? Even the oldest of the old-school providers seem to have crawled up the modern data stack since the data warehouse glory days, refocused on lakehouse + Open Table Formats either in their core platform or complementary products.

[ Parent | on HN ]

Re:Databricks and Neon by tqi on Wednesday May 14, 2025 @10:51PM (#43991377)

I'm curious - as someone who hasn't been paying super close attention to the space, what does it mean to be a full data platform? Is it just having all the different flavors of DB you might need all from one vendor, or is there also tighter integration than if you cobbled it together from multiple vendors?

[ Parent | on HN ]

Re:Databricks and Neon by Reebz on Wednesday May 14, 2025 @11:40PM (#43991618)

Essentially, yes. Different DB’s, federated queries (aka delta sharing, zero copy), definition/semantic layer tools, data engineering/pipelines, model training and notebooks, governance, data lineage, row/column/whatever access control.

It’s basically a luxury minivan. It’s may not be the fastest or prettiest or cheapest, but it’s a safe way for a large family of “data and AI people” to traverse a large organisation.

More seriously, I like to call it an “analytics workbench” in a professional setting.

[ Parent | on HN ]

Re:Databricks and Neon by te_chris on Thursday May 15, 2025 @06:42AM (#43993631)

Exactly, look at what GCP are adding to BigQuery.

[ Parent | on HN ]

Re:Databricks and Neon by adolph on Wednesday May 14, 2025 @07:15PM (#43990174)

> Not commoditising for enterprise. My last gig wouldn’t allow open source software or any company that might not be there in a decade, or which kept data anywhere but our own tenant.

Hence IBM talking up Iceberg: https://www.ibm.com/think/topics/apache-iceberg

[ Parent | on HN ]

Re:Databricks and Neon by jeffbee on Wednesday May 14, 2025 @07:55PM (#43990443)

> My last gig wouldn’t allow open source software or any company that might not be there in a decade

I bet they had VMware all over the place.

[ Parent | on HN ]

Re:Databricks and Neon by richardw on Thursday May 15, 2025 @03:25AM (#43992654)

Yup. And OpenShift. And Red Hat for Linux. And SAP. And IBM. But you know what? Tiny group of people relative to the impact, revenue and competitors. If we needed skills we clicked “buy”, 100 consultants would arrive who are experts, sort it and we’d move on. Not scratching around looking for people who know what we use and needing to learn 50 different open source tools. Coming from a much looser universe I learned to appreciate the principles for that context.

[ Parent | on HN ]

Re:Databricks and Neon by carlhjerpe on Thursday May 15, 2025 @11:00AM (#43995774)

I've worked at a Swedish streaming company, we bought a filesystem from IBM called "GPFS" or something before it was renamed. Well we had shit performance, reliability and everything else that could be bad. It was running on FC switches recommended by IBM, config from IBM, everything from IBM and it was an absolute fucking joke, metadata timed out regularly and the solution was "deal with it". I think my superiors should've been harder claiming we haven't received what we bought but we're Swedish so we'd rather just fall over and die.

[ Parent | on HN ]

Re:Databricks and Neon by richardw on Thursday May 15, 2025 @12:06PM (#43996440)

We only used MQ. Against my recommendation, but it was fine if imperfect, and a known entity to all concerned. Did not use that file system.

[ Parent | on HN ]

Re:Databricks and Neon by casey2 on Thursday May 15, 2025 @12:56PM (#43996936)

Battered woman syndrome

[ Parent | on HN ]

Re:Databricks and Neon by jflkdsjlcsuq on Thursday May 15, 2025 @12:27AM (#43991843)

I have been happily using ClickHouse for the past couple of years without any issue. Rock solid database with wide variety of features and fulfills all my needs. My favourite is the "external dictionary" feature which easily allows me to integrate it with other datastores like Postgres and Redis.

[ Parent | on HN ]

Re:Databricks and Neon by data_marsupial on Thursday May 15, 2025 @04:51AM (#43993103)

what do they use for ETL?

[ Parent | on HN ]