Show HN: MyDuck Server – Supercharge MySQL and Postgres Analytics with DuckDB

Re:Show HN: MyDuck Server – Supercharge MySQL and Postgres A... by robertclaus on Thursday November 28, 2024 @09:46PM (#42270352)

The current system diagram implies using duckdb's default storage format directly. I wonder how well this would actually work with the proposed zero-ETL design of basically treating this as a live replica. I was under the impression that as an OLAP solution DuckDB makes performance compromises when it comes to writes - so wouldn't live replication become problematic?

[ Parent | on HN ]

Re:Show HN: MyDuck Server – Supercharge MySQL and Postgres A... by fanyang01 on Thursday November 28, 2024 @10:12PM (#42270477)

Great question! Improving the speed of writing updates to DuckDB has been a significant focus for us. Early in the project, we identified that DuckDB is quite slow for single-row writes, as discussed in this issue: https://github.com/apecloud/myduckserver/issues/55

To address this, we implemented an Arrow-based columnar buffer that accumulates updates from the primary server. This buffer is flushed into DuckDB at fixed intervals (currently every 200ms) or when it exceeds a certain size threshold. This approach significantly reduces DuckDB's write overhead.

Additionally, we developed dedicated replication message parsers that write directly to the Arrow buffer, minimizing allocations.

A community contributor has validated & enhanced the effectiveness of our approach: https://github.com/apecloud/myduckserver/pull/207

We plan to publish benchmarks on replication latency and query performance in the coming weeks. Stay tuned!

[ Parent | on HN ]