> Second, the script we used provided both the "mark for deletion" capability ... (where recoverability is desirable), and the "permanently delete" capability that is required to permanently remove data when required for compliance reasons. The script was executed with the wrong execution mode and the wrong list of IDs. The result was that sites for approximately 400 customers were improperly deleted.
> To recover from this incident, our global engineering team has implemented a > methodical process for restoring our impacted customers.
[https://www.atlassian.com/engineering/april-2022-outage-upda...]
Anyone else find it disturbing that they are able to restore data that they deleted permanently for "compliance" reasons? If this is true, how were they ever compliant? I guess data is only permanently deleted when the engineering team is following their typical, non-methodical process...
Going back and purging things from the backups as part of the delete process would be overdoing it to a ridiculous degree.
That can mean as much as "you have to encrypt everything with a separate key, so that you can destroy the key for the given (say, personally identifiable) dataset making its retrieval irrecoverable"
I'm not saying that's the particular compliance reason they had here, or that the analysis you're giving is wrong, either. There is an interpretation where either of these ideas could be the correct one.
If this weren’t a concern, regulations would demand immediate deletion of data.
there's a whole lot of people in here who are way too quick to assume that just because one part of a permanent deletion process was inadvertently triggered and then caught while they still had backups, their whole permanent deletion process is a lie.
You seem to be right-ish, while the gdpr in certain circumstances allows you to keep backups of data that should have been deleted it seems like they are trying to discourage it in the future.
> ...It is, however, important to note that where data put beyond use is still held it might need to be provided in response to a court order. Therefore data controllers should work towards technical solutions to prevent deletion problems recurring in the future.
Also, modifying backups is a great way to inadvertently hose your backups.
It's your job to use technics which allow you to do this like using encryption on your backup and deleting the keys for it, for example.
Exactly. I'm not aware of any laws saying "you must delete this data immediately". More like "within X days or months". The permanently delete thing presumably skips some cooling-off period in the online database but not the backup, which seems perfectly appropriate, provided your backup retention is compliant.
Google has a nice page describing out their deletion process. [1] It doesn't go into product-specific technical details/steps (like marked as deleted within the product, row deleted from Bigtable/Spanner, major compaction guaranteed to happen, backups guaranteed to be deleted or unusable) but it says this:
Google> We then begin a process designed to safely and completely delete the data from our storage systems. Safe deletion is important to protect our users and customers from accidental data loss. Complete deletion of data from our servers is equally important for users’ peace of mind. This process generally takes around 2 months from the time of deletion. This often includes up to a month-long recovery period in case the data was removed unintentionally.
This is a best practice.
Delitio> It's your job to use technics which allow you to do this like using encryption on your backup and deleting the keys for it, for example.
If they'd thrown away the encryption key immediately, this would have been much worse. Instead of "we're down for 2 weeks?!?" (already quite bad) it'd be "our data is gone forever?!?". You never want to delete anything too quickly for exactly this reason.
[1] https://policies.google.com/technologies/retention?hl=en-US
> If a business stores any personal information on archived or backup systems, it may delay compliance with the consumer's request to delete, with respect to data stored on the archived or backup system, until the archived or backup system relating to that data is restored to an active system or next accessed or used for a sale, disclosure, or commercial purpose.
For example, if you delete an email or document on Google it moves to the "Trash" folder for 30 days.
When you manually empty the trash or the time window expires, most likely the next step would be a soft deletion for a few days where the data is still on hard drives but hidden from the application. Soft deletion is mainly protection against coding errors, since soft deletion is easy to undo if you've caused an incident but hard deletion (removing the data from disk) is not.
Then most likely a garbage collection process comes by a few days later and hard deletes the data from disk, leaving it only on tape backups
Finally, maybe a month or two later it disappears from the tape backups as they get rotated or otherwise disposed of
This addresses the needs of:
- Giving a good user experience (user "oops I made a mistake" undelete)
- Protecting against incidents due to coding errors (software engineer "oops I made a mistake" undelete)
- Making sure data disappears from both disk and backups within a certain time window, like maybe 30 or 60 days (comply with regulation and user expectations of data being cleared)
If you make backups, you are, almost by definition, unable to perform a full 'Compliance Delete' before the oldest backup in the set has expired.
Compliance-based deletion, if it is offered as a service, is almost always something time-based, like "we guarantee the data will be deleted 7 years from now". And then that deliberate deletion step is baked into the backup process.
So, i.m.o. at best they misrepresented the nature of the compliance deletion process. It never did what it was designed to do.
An overarching theme with these things is “legitimate business need” and “no indefinitely retained customer data. Having backups, system event logs, etc are all legitimate business needs. Based on the data type that business need may be days or years with things like financial and legal requirements.
Youre conflating permanent, immediate, and irrevocable. These are usually handled in different aspects. Think of accounts having multiple states like active, suspended, closed, terminated, purged. Some examples;
suspended: credentials/authnz immediately disabled, all data online, charges continue to accrue, can be restored in minutes.
Closed: credentials disabled, data online, processing stopped, charges stopped, may take manual intervention (hours) to return to active.
Terminated: creds & account irrevocably unavailable, online data deleted, offline data (backups) remains available.
Purged: all online and offline customer data irrevocably unavailable. This generally happens after a defined retention period for things like logs, backups, etc.
You can apply similar concepts to individual resources more granularly than the account.
Disclaimer: principal at AWS but the above is my own opinion/observation and does not represent my employer.
I'm still kinda salty about it. I understand why big services can't retain data indefinitely, but like... it's just a few KB of text, and that text happens to have a lot of sentimental value. Besides, OkCupid knows that I deactivated my account because I am a success story -- why not hold onto those profiles a bit longer? Or better yet, how about emailing an archive of those messages immediately when you click the "I'm leaving because I'm in a happy relationship now" button? /rant
If they had obtained the data without you supplying it freely then that would have been an entirely different matter, especially if it was used in ways that you did not consent to. But since that does not appear to be the case here the GDPR applies like it does to all data that is directly related to a data subject but continuing to store it on behalf of the user(s) that supplied it is not a problem.
Note that the user here is disappointed that their data which they consented to be kept is no longer there. This is a pretty clear indication that as far as they are concerned their expectation was the even with the GDPR up and running that such data would continue to be preserved as it is in almost every service that existed prior to may 2018.
It is precisely this kind of panicky thinking around the whole subject of the GDPR that gives these irrational responses, companies that suddenly no longer dare to mail you but you have to log in to their portal, which is secured by your email address and more of these totally weird constructs.
If they wanted to delete this data the better way would have been to positively contact the user (so that you know that they have received your message) to ask if their data should be deleted or not. That's good stewardship, just tossing it isn't.
But sure, we can break all of our users to avoid the possibility of you having to write some legal briefs and us paying a small fine for keeping data 7 days instead of three.
Compliance? The contract has expired, so there’s no legal basis for them to keep your data?