News for hackers. Stuff that matters.
Re:Databricks and Neon by ako on Wednesday May 14, 2025 @05:02PM (#43989186)
Is Iceberg involved in every read/write? I thought it was mostly metadata?
Re:Databricks and Neon by zhousun on Thursday May 15, 2025 @01:27PM (#43997264)
DataFile(parquet) is not enough for table with update/delete, (they are part of iceberg "metadata"). for CDC from OLTP use-cases, the pattern involves rapidly marking rows as deleted/ insert new rows and optimizing small files. This is required for minutes-latency replication.

And for second latency replication, it is more involving, you actually need to build layer on top of iceberg to track pk/ apply deletion.