News for hackers. Stuff that matters.
Re:Moving from relational data to events by AtlasBarfed on Saturday December 16, 2023 @11:02AM (#38665189)
Four years ago I heard "Kafka IS your database".

I thought maybe these insane people (probably parroting some tech company enterprise penetration propaganda e.g. Confluent) would have a better story, but... no.

Anyway, yeah, sure, keep logs. But a lot of that article about commands and events is something that only exists if you had one ubiquitous language, system, and OS. You know, the almost literal "seamless" where there aren't any seams.

Sure that will probably plug into some enterprise bus and enterprise integration and enterprise ... anyway.

Competent developers will understand what events to preserve and log and possibly allow retry/repeats.

Anyone who has looked at a Splunk bill will realize that just storing all the logs everywhere is very expensive, which is another way of saying "wasteful". But any generic enterprisey event system will basically start and end with splunk-level log aggregation and kinda-analysis.

Re:Moving from relational data to events by dgellow on Saturday December 16, 2023 @01:13PM (#38666451)
Dealt with something like this when I joined a previous job years ago, Kafka was the main data store, microservices had their state in memory they would build at startup by processing their whole history of events (that was ridiculously slow for some of them). Then GDPR came in place and the whole “keep everything in Kafka forever” had to face the reality of “not allowed to keep PII for longer than 30 days” :)

(Before someone suggests it, no messages weren’t encrypted, just throwing the key away wasn’t an option)

Re:Moving from relational data to events by andsbf on Saturday December 16, 2023 @03:37PM (#38667451)
> ... they would build at startup by processing their whole history of events (that was ridiculously slow for some of them)

There is a basic technique to solve this, you snapshot every "nth" event

Re:Moving from relational data to events by jabradoodle on Saturday December 16, 2023 @01:24PM (#38666526)
Would be expensive but you can do a copy and replace to keep the data you need, on a new topic.

If you don't need the data, then you don't need it.