News for hackers. Stuff that matters.
Re:Event Sourcing (2017) by xtagon on Wednesday January 01, 2020 @07:55PM (#21932031)
Great article. One thing that wasn't pointed out that might be of interest to someone learning about Event Sourcing is that it introduces some challenges if you are to be compliant with GDPR and similar laws. For example, if your event log is immutable, and you use it as an audit log, then by nature you are not ever deleting data. There are solutions to this (for example, crypto-erasure), but it can be non-trivial to implement.
Re:Event Sourcing (2017) by tunesmith on Wednesday January 01, 2020 @07:58PM (#21932042)
I saw the Akka people talking about this on twitter once, I think they were theorizing that encrypting the data in the log would be sufficient, because then "deleting the key" could be interpreted as the deletion of the record (even though the useless data still exists in the log). But I'm not sure this was ever legally validated?
Re:Event Sourcing (2017) by xtagon on Wednesday January 01, 2020 @09:47PM (#21932568)
That is what I was referring to as "crypto-erasure" (also known as "crypto-shredding"). I'm not sure what counts as legally validated, but some have shared concerns that the encryption you use to do this would need to be future-proof against, say, advancements in quantum computing cracking the encryption down the road even after the key has been thrown out.
Re:Event Sourcing (2017) by rodocite on Wednesday January 01, 2020 @10:23PM (#21932750)
if the references don't point to actual data, you don't need this.

- minimum 2 parts. a relay (reference hash) and the cold/true storage portion. you can break the reference hash up and reassemble only for secret holders. which brings us to:

- content-based routing

- there are also deterministic vaults for rolling keys

i'm actually not sure in what high-level situation "crypto-erasure" would work in because being able to re-key a reference means you have complete control. so why would you need to erase the "bad" key when you can just switch the reference?

Eth draft for enabling cold storage relay https://github.com/ethereum/EIPs/blob/master/EIPS/eip-1077.m...

Decentralized ID https://www.w3.org/TR/did-core/

^ both generally use the same concept i described above and solve GDPR "delete" issue. actually, it solves GDPR completely if you can just rely on the DID. "hard delete" is a separate issue, though-- no other way to get around that but to fork/version + replay your store and re-reference anyone who wants to hard delete if you didn't use reference hashes.

Re:Event Sourcing (2017) by guywhocodes on Wednesday January 01, 2020 @09:07PM (#21932393)
That is a very interesting approach, however I can imagine that it can be quite a large database of encryption keys.

I'll have to build a small system for this and try it.

Re:Event Sourcing (2017) by imglorp on Wednesday January 01, 2020 @09:40PM (#21932535)
Oh, why? Holding one key for every human on earth would fit in 8 TB plus read-mostly replicas. You'd delete their key if they made GDPR removal request and the keyless, encrypted, immutable entries would be entombed in place.
Re:Event Sourcing (2017) by pookeh on Thursday January 02, 2020 @12:14AM (#21933197)
What do you do when the data involves multiple people?
Re:Event Sourcing (2017) by shrimpx on Thursday January 02, 2020 @03:58AM (#21933896)
It's unclear (regardless of crypto shredding) what the delete policy should be if the data involves multiple people.
Re:Event Sourcing (2017) by randolorian on Thursday January 02, 2020 @10:05AM (#21935872)
Why would it? Typically you would have one stream per user.
Re:Event Sourcing (2017) by tunesmith on Thursday January 02, 2020 @01:42PM (#21938296)
Event: "User A sent Message X to User B". (I think the answer here is again to have PII be in other repositories, and only refer to them by id in the log.)
Re:Event Sourcing (2017) by specialist on Thursday January 02, 2020 @02:17PM (#21938673)
Good question. Create alias proxies?
Re:Event Sourcing (2017) by specialist on Thursday January 02, 2020 @02:13PM (#21938633)
Just like password stores. Store salted and hashed password value, vs actual password.

Cleverly applying this strategy to all sensitive datas is the Translucent Database thesis.

Re:Event Sourcing (2017) by eweise on Thursday January 02, 2020 @01:01AM (#21933325)
Don't stick PII in the events. Write it to a dedicated PII db and reference it in the event. Easy to get rid of the person's data. Just delete from the PII db.
Re:Event Sourcing (2017) by rodocite on Wednesday January 01, 2020 @08:16PM (#21932139)
I'm pretty sure you can "soft-delete" for GDPR compliance. So this concern is sort of a non-issue.

Besides that:

1. If you're using an immutable structure, as long as you use references, you can obfuscate data. Blockchains ran into this problem before GDPR requirements and that's essentially all they do.

2. ^ "Update" strategy for event sourcing is the same as above. Essentially a copy of the log or the log slice then a re-indexing to remove/update streams, events, projections, etc. Greg Young talks about the indexing internals (for Event Store) in the 2012 video.

Re:Event Sourcing (2017) by nchase on Wednesday January 01, 2020 @10:08PM (#21932691)
Can you describe more precisely how you're defining "soft-delete" here?
Re:Event Sourcing (2017) by rodocite on Wednesday January 01, 2020 @10:27PM (#21932779)
data is still stored somewhere but any routing to the data is disabled and the entity is disassociated.
Re:Event Sourcing (2017) by GordonS on Thursday January 02, 2020 @08:12AM (#21935063)
Could you point towards an authoritative source for this claim?

As a consumer, if I request deletion of my data, I expect it to be actually deleted - not just have a "deleted" flag set.

With soft-deletion, the data is still right there, ready to be abused after a breach.

Re:Event Sourcing (2017) by lawik on Thursday January 02, 2020 @02:06AM (#21933535)
Soft delete in the sense of removing availability but keeping the data shouldn't be GDPR compliant from what I've seen. Not in regards to right to be forgotten and restrictions on keeping PII.
Re:Event Sourcing (2017) by the_gipsy on Thursday January 02, 2020 @11:15AM (#21936594)
"soft-delete" is not GDPR compliant.
Re:Event Sourcing (2017) by codebje on Thursday January 02, 2020 @12:44AM (#21933277)
Your event log should IMO be nominally immutable rather than actually immutable.

You should feel free to take actions such as expunging private or sensitive data as appropriate. Keep the events, but rewrite them to contain only the desired data. Trivial to implement, and simple.

I'd only worry about stuff like crypto-erasure if you physically cannot alter the past, such as if you have a requirement for non-repudiation or some such. Doing it just for technical purity isn't worth the cost :-)

Re:Event Sourcing (2017) by andreareina on Thursday January 02, 2020 @02:21AM (#21933575)
I think cases where you cannot alter the past (or can only do so with difficulty) are fairly common; backups will easily tend to fall in that category.
Re:Event Sourcing (2017) by TheCoelacanth on Friday January 03, 2020 @01:55PM (#21949056)
Backups have the same GDPR concerns regardless of whether you are doing event sourcing or not.