The really hard part is dealing with the data still in backups.

It leads to having to do crazy things like individual keyed encryption per user, escrow hilarity, etc.

I do a lot of work with Kafka, GPDR removal requests for data that's still stored in a Kafka topic is so much fun.

I worked at a large company as it was implementing GDPR compliance, and I can attest that it was a nightmare of drudgery. I think it's interesting that our entire tech stack is designed to assume the business has a right to keep user data for as long as they want (forever), and compliance with GDPR delete requests is like a one-off exception implemented at a very sketchy redaction task that has to be careful not to leave broken foreign keys and such around.

It's a nightmare to do now, but our tech needs to change such that we assume data might be a hot potato that you need to be able to get rid of, and not some immutable asset that you can hang onto forever.

Barring more foundational and revolutionary changes in our tech stacks, another poster commented that a best practice is to have tests that delete user accounts and all associated records without breakage and to build these tests into your test suite from the very inception of the project.

Far from a magic cure, but by building it into the app from inception, it seems more possible.

Thanks for that insight. It's an interesting solution to use per-user encryption for that. Don't you have the same problems all over again with respect to preserving access to user specific data? I.e. needing key backups for business continuity, but needing to wipe those keys for deletion requests.

Per user encryption doesn't work in real systems. These have been implemented for several decades and no one uses them because they have pathologically terrible scalability and performance. The simple act of requiring encryption of individual user data forces the use of extremely suboptimal data structures relative to what would be used in the non-encrypted case.

A single AES encryption block is 16 bytes; internal storage of records is managed in terms of individual bits if you don't encrypt them. You also can't compress data encrypted in this way, which is table stakes for databases. The encryption key schedule alone for a single record will be much larger than the typical record! Requiring a per-user encryption key has the unfortunate effect of bloating database storage and memory requirements by at least 10x, while massively reducing computational throughput.

The design of delete-efficient database kernels is an interesting unsolved problem in computer science. No one knows how to do it. People that think there are trivial solutions have not studied the problem adequately. It hasn't been a priority to solve this problem, but its reputation for being theoretically difficult is well-deserved.

"I've not seen this in real systems, therefore it can't exist" ?

Or...just have a documented retention policy for backups and delete them after a certain timeline.

Guess we’ll find out when a leaks happen, and companies start getting fines.

GDPR does differentiate between structured data (I believe it uses the term “identifiable records” or similar), and huge heap of unstructured data where an individuals data can’t be quickly retrieved as it’s own atomic unit. With much stricter requirements for anything structured.

So for data on a HDD that could be recovered, but is an unstructured mess. You’re probably ok, as long as you took reasonable steps to protect the data, like full disk encryption (which could occurs below the file system, and thus allow you to recover data deleted from the FS with a full disk scan, as long as you still had the keys). If you just had peoples data unencrypted on a HDD, and didn’t securely erase it before dispose, then that’s probably still a GDPR violation.

What does GDPR say about obscuring data?

Instead of hard-deleting a structured record for Bob Smith, can you leave the record intact and scrub it of identifying data so that I don't e.g. break all of the records for orders that Bob Smith made?

> Instead of hard-deleting a structured record for Bob Smith, can you leave the record intact and scrub it of identifying data so that I don't e.g. break all of the records for orders that Bob Smith made?

The various laws (GPDR, and similar ones in my jurisdiction) abuse the hell out of vague descriptions like "personal identifying data".

For example, scrubbing a customer record of all names, phone numbers, contact info, etc and leaving only the ID in the record still makes it possible to personally identify the person by:

1. Looking through the orders records to build up a profile depending on what they order,

2. Then using the delivery records from the courier company to get an address,

3. Then using existing public information to repopulate all the fields in the customer record.

In any given business, no matter how big or small, simply removing all references to a customers personal information will still leave the customer personally identifiable. This process can even be automated.

You can as long as you delete all the personal data of the user and make sure that other remaining data is anonymized in a way that you can no longer associate it with the user.