We don't delete events, just replace sensitive info in events with the specific string marker (by direct update in DB) and then update projections. In this case, the system remains consistent because all the related aggregates and events are still there. You can safely rebuild all the projections without issues and see these string markers instead of deleted info.
^ 100% this. Do not delete events. They are your source of truth - you shouldn't really even modify them but stripping out PII is "alright".
Re 24M+ records: create a batch runner that goes through "jobs" to perform stripping/cleaning tasks. To store state (and to organize cleaners), use a distributed store such as etcd - that way you can bookmark where you were at in the cleaning process.
Two paths : encrypt customer data in payloads with a customer-specific key that you can throw away at will ; OR ; allow some events to rewrite their history / stream.
Re:Event Sourcing (2005)
by ComodoHacker on Thursday August 12, 2021 @02:50AM
(#28152873)
Not an advocate, but you just run the script to clean the data (as ordinary events).
It's not really different than handling regular backups. You don't wipe them, just guard them well until they expire.
Does it comply when I cant see the data but it is still there?
I inherited an application with 24M+ events and I have no idea how to go about it.