News for hackers. Stuff that matters.
Re:Moving from relational data to events by dgellow on Saturday December 16, 2023 @01:13PM (#38666451)
Dealt with something like this when I joined a previous job years ago, Kafka was the main data store, microservices had their state in memory they would build at startup by processing their whole history of events (that was ridiculously slow for some of them). Then GDPR came in place and the whole “keep everything in Kafka forever” had to face the reality of “not allowed to keep PII for longer than 30 days” :)

(Before someone suggests it, no messages weren’t encrypted, just throwing the key away wasn’t an option)

Re:Moving from relational data to events by andsbf on Saturday December 16, 2023 @03:37PM (#38667451)
> ... they would build at startup by processing their whole history of events (that was ridiculously slow for some of them)

There is a basic technique to solve this, you snapshot every "nth" event

Re:Moving from relational data to events by jabradoodle on Saturday December 16, 2023 @01:24PM (#38666526)
Would be expensive but you can do a copy and replace to keep the data you need, on a new topic.

If you don't need the data, then you don't need it.