Re:Understanding Parquet, Iceberg and Data Lakehouses
by jamesblonde on Saturday December 30, 2023 @12:52AM
(#38813077)
I agree with all of this. Databricks are also holding back features from open source Delta (like bloom filters), which is their right. But then you can't claim it is a community-driven open format, unless it is an animal farm version of that, where one of the versions is the Pig (some are more equal than others).
[ Parent
| on HN ]
Re:Understanding Parquet, Iceberg and Data Lakehouses
by hacful-tonteg on Saturday December 30, 2023 @03:59AM
(#38813806)
Databricks has a lot of nice closed-sourced components, e.g., Unity Catalog, Delta Live Tables and Photon (a C++ implementation of Spark).
Delta itself seems fairly open-source: https://github.com/orgs/delta-io/projects/10/views/1 and hopefully someone will implement Liquid Clustering!