Show HN: BemiDB – Postgres read replica optimized for analytics

Re:Show HN: BemiDB – Postgres read replica optimized for ana... by nitinreddy88 on Thursday November 07, 2024 @09:17PM (#42083460)

How does update or continuous inserts get written/updated to parquet files? Architecture doesn't show nor anything in docs.

1. All the benchmarks/most of the companies, show one time data exists and try querying/compressing in different formats which is far from reality

2. Do you rewrite parquet data every time new data comes? Or partitioned by something? No examples

3. How does update/delete works. Update might be niche case. But deletion/data retention/truncation is must and I don't see how you support that

[ Parent | on HN ]

Re:Show HN: BemiDB – Postgres read replica optimized for ana... by exAspArk on Thursday November 07, 2024 @10:38PM (#42083887)

Our initial approach is to do full table re-syncs periodically. Our next step is to enable incremental data syncing by supporting insert/update/delete according to the Iceberg spec. In short, it'd produce "diff" Parquet files and "stitch" them using metadata (enabling time travel queries, schema evolution, etc.)

[ Parent | on HN ]