Re:Event Sourcing (2005)
by const314 on Thursday August 12, 2021 @07:32AM
(#28154937)
We don't delete events, just replace sensitive info in events with the specific string marker (by direct update in DB) and then update projections. In this case, the system remains consistent because all the related aggregates and events are still there. You can safely rebuild all the projections without issues and see these string markers instead of deleted info.
[ Parent
| on HN ]
^ 100% this. Do not delete events. They are your source of truth - you shouldn't really even modify them but stripping out PII is "alright".
Re 24M+ records: create a batch runner that goes through "jobs" to perform stripping/cleaning tasks. To store state (and to organize cleaners), use a distributed store such as etcd - that way you can bookmark where you were at in the cleaning process.