Ask HN: What are you working on? (March 2025)

Re:Ask HN: What are you working on? (March 2025) by arcbyte on Sunday March 30, 2025 @07:41PM (#43528971)

I'm working on a new Event Sourcing database that elevates the WAL into a first class application concept like a message queue. So instead of standing up a postgresql instance and a kafka instance and a bunch of custom event sourcing plumbing, you stand up this database and publish all your application events as messages. For the database part you just define the mappings from event to table row and you get read models and snapshots for free.

The real key here is how migrations over time are handled seamlessly and effortlessly. Never again do you have to meet with half a dozen teams to see what a field does and if you still need it - you can identify all the logic affecting the field and all the history of every change on the field and create a mapping. Then deploy and the system migrated data on the fly as needed.

Still in stealth mode and private github but the launch is coming.

[ Parent | on HN ]

Re:Ask HN: What are you working on? (March 2025) by NathanFlurry on Sunday March 30, 2025 @08:12PM (#43529189)

Neat! How does this differ from KSQL?

[ Parent | on HN ]

Re:Ask HN: What are you working on? (March 2025) by arcbyte on Sunday March 30, 2025 @11:01PM (#43530429)

It was inspired by KSQL! But really KSQL isn't a database, it just gives you some sql processing. Nobody stands up Kafka and KSQL when they want database of any kind.

The difference is that I'm building a database and exposing the WAL to the application layer. What that means is that you can connect your legacy DB application and have it issuing insert and update queries which are now native messages on a distributed message queue. Suddenly you gain a conokete audit trail for your entities without brittle tables and triggers to track this. Then instead of hooking up Qlik or devezium, you can just stand up another instance of the DB, propagate the realtime WAL and you've got streaming analytics - or whatever new application you want.

[ Parent | on HN ]

Re:Ask HN: What are you working on? (March 2025) by Benjamin_Dobell on Sunday March 30, 2025 @08:06PM (#43529159)

Interesting. Sounds like a similar premise to Supabase Realtime. I'll keep an eye out.

[ Parent | on HN ]

Re:Ask HN: What are you working on? (March 2025) by sanderjd on Sunday March 30, 2025 @07:49PM (#43529038)

Sounds very useful! Looking forward to seeing how it turns out!

[ Parent | on HN ]

Re:Ask HN: What are you working on? (March 2025) by kukkeliskuu on Sunday March 30, 2025 @09:52PM (#43529884)

I have done similar for a few customers. I have found that useful pattern is to have both raw queues (incoming data) and clean queue (outgoing data). Outgoing data in single queue only (so all changes are ordered, so we avoid eventual consistency) that has well-defined data model (custom DSL for defining it) and tables/REST api that corresponds 1-to-1 to the data model. Then we need mappings from raw queues to the clean queue.

[ Parent | on HN ]

Re:Ask HN: What are you working on? (March 2025) by arcbyte on Sunday March 30, 2025 @10:54PM (#43530382)

Interesting experience! Can you explain a bit more about the raw queues vs clean queues? Is it literally just incoming and outgoing queues or was there a problem that you were trying to solve?

[ Parent | on HN ]

Re:Ask HN: What are you working on? (March 2025) by kukkeliskuu on Sunday March 30, 2025 @11:33PM (#43530617)

The raw data was whatever came from the individual source systems. Mostly these were external systems. As we are talking large organizations, there are always various kinds of problems with the source data. So the aim was to bring it all together under a single, shared data model. It was very helpful to use internal DSL for defining clean models, we called it the canonical data model.

[ Parent | on HN ]

Re:Ask HN: What are you working on? (March 2025) by dm3 on Monday March 31, 2025 @06:33AM (#43533319)

What would be the largest difference to Kurrent (former EventStore)?

[ Parent | on HN ]