Re:Databricks and Neon
by datadrivenangel on Wednesday May 14, 2025 @01:32PM
(#43987143)
But why would you buy an operational DB from Databricks? The only thing that makes sense is Databricks flailing to maintain market cap.
[ Parent
| on HN ]
In addition to the AI use cases, sometimes you wanna share the data warehouse data in oltp way for fast lookups and high concurrency. Not sure whether Neon will do that but I hope so.
One example from Snowflake is hybrid tables which adds rowstore next to columnar.
OLAP + OLTP = HTAP
SingleStore been doing that for years . Unistore been strugglin
I wonder why Singlestore has been so unpopular (at least I never hear about it). Quick guess is that HTAP itself isn’t a significant feature requirement, maybe just a cherry on top of other major db features.
Re:Databricks and Neon
by orangechairs on Wednesday May 14, 2025 @04:26PM
(#43988808)
SingleStore (nee MemSQL) was/is niche -- great fit inside banks in particular.
Re:Databricks and Neon
by moonikakiss on Wednesday May 14, 2025 @04:35PM
(#43988881)
it was a couple things.
1. It's really really hard to replace anyone's OLTP.
2. OLTP and OLAP are owned by such different teams. Who do you make your champion?
3. The modern HTAP dream is possible without something like SingleStore. You need a columnstore that can keep up with your OLTP tables and provide transactional correctness. Who cares if it's all within one system
ps: I worked at SingleStore. https://www.mooncake.dev/blog/htap-is-dead
ETL to bring all your data into Databricks/Snowflake is a lot of effort. Much better if your OLTP data already exists in Databricks and you directly access it from your OLAP layer.
Re:Databricks and Neon
by moonikakiss on Wednesday May 14, 2025 @03:56PM
(#43988553)
With the push towards open table formats (Iceberg) from both Snowflake and Databricks, it's even harder to get your Postgres OLTP tables ready for OLAP.
The problem isn't in the CDC / replication tools in the market.
The problem is that columnar stores (especially Iceberg) are not designed for the write /upserts patterns of OLTP systems.
They just can't keep up...
This is a big problem we're hoping to solve at Mooncake [0]. Turn Iceberg into an operational columnstore. So that it can be keep up (<s freshness) with your Postgres.
Is Iceberg involved in every read/write? I thought it was mostly metadata?
DataFile(parquet) is not enough for table with update/delete, (they are part of iceberg "metadata").
for CDC from OLTP use-cases, the pattern involves rapidly marking rows as deleted/ insert new rows and optimizing small files. This is required for minutes-latency replication.
And for second latency replication, it is more involving, you actually need to build layer on top of iceberg to track pk/ apply deletion.