Aren't the alternatives you mentioned - icerberg and duckdb - both storage solutions while spark is a way to express distributed compute? I'm a bit out of touch with this space, is there a newer way to express distributed compute?
Re:Databricks and Neon
by Nate75Sanders on Wednesday May 14, 2025 @11:48AM
(#43985954)
Flink. It has more momentum than Spark right now.
"momentum" is a tricky word. Zig has more momentum than C++, but will it ever overtake the language? I'd bet not.
Re:Databricks and Neon
by franktankbank on Thursday May 15, 2025 @09:37AM
(#43994968)
Well its not a tricky word it just wrong. Velocity maybe. Or more probably acceleration.
Re:Databricks and Neon
by briankelly on Wednesday May 14, 2025 @05:47PM
(#43989545)
Flink is designed around streaming first, while Spark is built around batch first and you're likely best off selecting accordingly. Though any streaming application likely needs batch processing to some degree. Latency vs throughput.
Re:Databricks and Neon
by tomjakubowski on Wednesday May 14, 2025 @01:15PM
(#43986966)
DuckDB is not only a storage solution. It can directly query a variety of file formats at rest, without having to re-store anything. That's one of its selling points: you can query across archival/log data stored in S3 (or wherever) without needing to "ingest" anything or double-pay to duplicate the data you've already stored.
Re:Databricks and Neon
by steve_adams_86 on Thursday May 15, 2025 @01:57AM
(#43992239)
I’m just getting into DuckDB lately and finding this feature so exciting. It’s a totally new paradigm. Such a great tool for scientists, and probably many other people. I wish I took it seriously sooner.
Re:Databricks and Neon
by mritchie712 on Wednesday May 14, 2025 @01:28PM
(#43987115)
duckdb is primarily a query engine. It does have a storage format, but one of it's strengths is querying data where it already resides (e.g. a parquet file sitting in S3).
There are some examples[0] of enabling DuckDB to manage distributed workloads, but these are pretty experimental.
Thanks for the pointers!
Not a new way like Ray, but a new way to express Spark super-efficiently (GPU-acceleration): https://news.ycombinator.com/item?id=43964505
Re:Databricks and Neon
by robertlacok on Thursday May 15, 2025 @05:45AM
(#43993375)
I think what many people are finding out is they don’t really need distributed processing. DuckDB on a single node can get you really far, and it’s much simpler.
The product is still centered Spark, but most companies don't want or need Spark and a combination of Iceberg and DuckDB will work for 95% of companies. It's cheaper, just as fast or faster and way easier to reason about.
We're building a data platform around that premise at Definite[0]. It includes everything you need to get started with data (ETL, BI, datalake).
0 - https://www.definite.app/