Re:Databricks and Neon
by ako on Wednesday May 14, 2025 @05:02PM
(#43989186)
Is Iceberg involved in every read/write? I thought it was mostly metadata?
[ Parent
| on HN ]
DataFile(parquet) is not enough for table with update/delete, (they are part of iceberg "metadata").
for CDC from OLTP use-cases, the pattern involves rapidly marking rows as deleted/ insert new rows and optimizing small files. This is required for minutes-latency replication.
And for second latency replication, it is more involving, you actually need to build layer on top of iceberg to track pk/ apply deletion.