moonikakiss

++

I did due-diligence on Builder.AI for a venture firm I was interning at (circa 2019). It was extremely apparent (Glassdoor, talking to any employee) it was complete BS.

When I say apparent, it took less than 15 minutes and a couple of google searches to get a sniff of it.

Somehow, you can still raise $500MM ++.

I think about that a lot

totally agreed on 3. You're also missing the challenges of dealing with updates/deletes; and managing the many small files.

CDC from OLTP to Iceberg is extremely non-trivial.

yep exactly. we can wait for replay LSN. So you're only reading once all writes to OLTP are complete.

great point.

with pg_mooncake v0.2 (launching in ~couple weeks), you'll be able to get a columnar copy of your Postgres that's always synced (<s freshness).

Keep your write path unchanged, and keep your Postgres where it is. Deploy Mooncake as a replica for the columnar queries.

yes you can. Even if the columnar tables are in the read replica. you'll be able to do joins with your OLTP tables

exactly. we take the CDC output / logical decoding from your OLTP tables and write into a columnar format with <s freshness.

We had to design this columnstore to be 'operational' so it can keep up with changing oltp tables (updates/deletes).

You'll be able to deploy Mooncake as a read-replica regardless of where your Postgres is. Keep the write path unchanged, and query columnar tables from us.

--- v0.2 will be released in preview in ~a couple weeks. stay tuned!

great blog. It seems like you might benefit from columnar storage in Postgres for that slow query that took ~20seconds.

It's interesting that people typically think of columnstores for strict BI / analytics. But there are so many App / user-facing workloads that actually need it.

ps: we're working on pg_mooncake v0.2. create a columnstore in Postgres that's always consistent with your OLTP tables.

It might help for this workload.

Even more so after Neon + Databricks.

it was a couple things. 1. It's really really hard to replace anyone's OLTP. 2. OLTP and OLAP are owned by such different teams. Who do you make your champion? 3. The modern HTAP dream is possible without something like SingleStore. You need a columnstore that can keep up with your OLTP tables and provide transactional correctness. Who cares if it's all within one system

ps: I worked at SingleStore. https://www.mooncake.dev/blog/htap-is-dead

This is really, really exciting. I see it as the “right” way OLTP and OLAP will converge.

The OP and I built an HTAP system at SingleStore. A single database with one copy of data for both OLTP and OLAP workloads. HTAP never took off [0].

What we learned was that OLTP (Postgres) should handle OLTP, while OLAP (data warehouses/lakes) should handle OLAP, with replication between them.

Designing the 'up-to-date' replication between these systems is hard.... columnar stores just aren’t built for OLTP‑style writes, and can't keep up with your OLTP tables.

Let’s see if Databricks and Neon can pull this off

“give me up‑to‑date Postgres tables in Unity Catalog", no debezium --> kafka --> flink --> Iceberg. With Spark jobs in the back ensuring that Iceberg is an optimal state.

https://www.mooncake.dev/blog/htap-is-dead

With the push towards open table formats (Iceberg) from both Snowflake and Databricks, it's even harder to get your Postgres OLTP tables ready for OLAP.

The problem isn't in the CDC / replication tools in the market.

The problem is that columnar stores (especially Iceberg) are not designed for the write /upserts patterns of OLTP systems.

They just can't keep up...

This is a big problem we're hoping to solve at Mooncake [0]. Turn Iceberg into an operational columnstore. So that it can be keep up (<s freshness) with your Postgres.

https://www.mooncake.dev/