News for hackers. Stuff that matters.
Re:Understanding Parquet, Iceberg and Data Lakehouses by jamesblonde on Saturday December 30, 2023 @12:52AM (#38813077)
I agree with all of this. Databricks are also holding back features from open source Delta (like bloom filters), which is their right. But then you can't claim it is a community-driven open format, unless it is an animal farm version of that, where one of the versions is the Pig (some are more equal than others).
Re:Understanding Parquet, Iceberg and Data Lakehouses by hacful-tonteg on Saturday December 30, 2023 @03:59AM (#38813806)
Databricks has a lot of nice closed-sourced components, e.g., Unity Catalog, Delta Live Tables and Photon (a C++ implementation of Spark).

Delta itself seems fairly open-source: https://github.com/orgs/delta-io/projects/10/views/1 and hopefully someone will implement Liquid Clustering!