In banking and bookkeeping, there’s no such thing as a “delete”. Once something is in the ledger you can’t undo it - you have to make a new entry that negates the old one.

Yes, but banks tend to have websites with accounts and those accounts need to be deactivated when a customer should no longer have access (or, even more finicky, specific accounts for a client need to be deactivated or activated as they change their usage).

All this essentially forces the use of some sort of soft deletion. (Activation flags are sort of just a more complicated form of soft deletion).

Status (active/inactive/other) and existence (present or deleted) are very different things, they may be separated most of the time. Legal requirements can prevent deletion of inactive data, so in some cases one may have a lot of inactive but must keep data. Soft delete can help in some scenarios, for example you want to give people a way to re-activate accounts, but hide the ones marked as deleted.

not every thing is a transaction though, say a deleted piece of ref data that still has FKs to other stuff etc

These are still versioned in many serious finance systems

Is the same true in mass surveillance?

and the blockchain.

Which is a decentralized, distributed ledger, so... banking! :)

By that definition the tally marks on my beer mat are banking, too.

"banking"

I would argue that in many cases the concept behind soft deletion is to make deletion permanent.

Hard deletes retain no memory of what you wanted to be gone, so any malfunctioning sync process will continuously recreate the deleted record soon after it's deleted. Soft deletes are often the only way to make sure deleted records don't reappear.

Couldn't a malfunctioning sync process undo a soft delete as well?

Null is more ambiguous than an explicit conflict. If there is literally no record, even of the delete action then there's no timestamp for last write wins.

Assuming you're using a modern database, replication is done with paxos/raft and they are formally proven to not allow this to happen as both edits/deletions are both just entries in distributed event log.

My experience at a few start-ups has been that account deletion just isn't prioritized. It's not a focus when building an MVP. If the application ever gains traction, everyone is then terrified they'll accidentally delete customer data that they never delete anything. It's a shame. As a user, when I delete my account or data in my account, I want you to permanently delete it, not keep it around and just make inaccessible to me.

If it makes you feel better, the startups that don't have time to delete your data probably don't have a viable disaster recovery plan either.

As we learned from the Atlassian snafu, even giant companies with billions in revenue often can't recover from disasters. (I try to test mine every 6 months. I've never had a test go perfectly.)

Good to know. What do you were tests look like?

I've also seen businesses retain "deleted" data in order to support legitimate data analysis work in the future. And it actually can help significantly. Maybe scrubbing PII from deleted accounts is a good idea, but those deleted accounts are perfectly good data points, especially in smaller/newer companies with lower-volume data streams.

If you want to retain anonymous statistics, fine. I'm not keen on you retaining my actual data so you can monetize it after our business relationship has concluded. The biggest thing a small team can learn is why they failed to retain a customer or a trial that didn't convert. For that, usage metrics are considerably more valuable than user data.

The business owns your data. Why would they delete it if they don't have to?

Why not just use an audit table, to keep from littering your indices?

An audit table is a similar, but not entirely equivalent tool. There are circumstance where both audit tables / soft deletes are appropriate, and where only one of the two is. Other systems that work are append-only tables, event driven systems, and I'm sure there are more that I'm not aware of.

You still need a record of the data, which probably isn't fully captured in the audit table

Then there's always the joy of a situation where your client is being sued by one of their clients and now needs your help recovering everything possible from your platform. And you're going to help them because you'd like to keep them as a client rather than let them be sued into oblivion.

Ha, and then there is the opposite regulation that you have to delete user data.

Well you see, you need a complete record of when it was created, every change that occurred, everyone who could view it and also log every access attempt. But it. You're not actually supposed to keep it. Just everything surrounding it.

Not quite. GDPR (and equivalents) have clear escape hatches to allow you to store data if you have good reason (even if the data subject requests its removal).

Invoices, from the article, is a great example. That record must remain unchanged in most financial regulations. I’d wager a customer sending a deletion request for invoices will be met with raucous laughter from the legal and finance teams.

In some industries the retention regulations trump the deletion ones. I believe finance is one area where they will delete some of your data, but are still required to maintain 7 years of specifically listed data.

This is almost certainly going to bite you if you don't push all customer identification data out of your main data stores.

And it will go a long way to making your services harder to use if you don't allow users to associate friendly names with things. And to assume that the same friendly name will be used for a future item. (For example, if you name devices based on the room you put them in. Is reasonable to think that when you replace a device, that you are likely to want to reuse the name.)

And yet another part is making deletes (appear) instantaneous: useful when it involves cleaning up a bunch of "related" data possibly living on different services (eg. S3, ES...).

This also helps with the original goal of making them safer by manually implementing "eventual consistency" for data living outside the transactional world.

Don't make deletes appear instantaneous? If you have heavy weight systems, then it makes sense for provisioning and deleting entities is a process, that should be open to monitoring.

Exposing details of long lived background processes, and especially deletion processes to users (who in many cases couldn't care less) is a lot of work that's probably not worth it for the rare "hey, something went wrong" case — it's usually perfectly acceptable to raise that with the support team and let them use internal tools to debug and investigate.

I am sure there are cases where it is worthwhile, but most of the times I've hit this in common web apps, it wasn't.

I disagree. Any system that doesn't transition an entity to "deleting" is almost certainly going to be a pain point for me when I go to delete something.

This can be seen as related to soft deletes. But I consider it more marking intent in the system. Lets you make "delete" a simple field update that will be carried out by the backend.

Say you are attempting to delete a tweet you made: you click on the delete button, and you want to see that your tweet is in the process of being deleted, but not really deleted yet (as the platform goes and updates all the retweet references and such)?

Really? What value do you as a user get from knowing the details of this transitory state? Do you also want to see a progress bar of how many references have been updated, what's the progress of evicting the tweet from search indexes, etc? What's wrong with all this appearing instantaneous and actual delete happening in the background over 5 minutes or 5 hours?

It seems you are in favour of this approach, just don't want to call it "soft delete" (since it may be followed by a hard out-of-band delete).

I'd rather see acknowledgement that it is having to do more work, than be surprised when it still shows up on another timeline for a bit longer.

I don't necessarily care on a progress bar, as those typically have their own woes.

And don't get me wrong, there are easy deletion cases that I am ok with appearing to happen immediately. I just find hiding processes from me is usually more annoying than it is worth. Worse, when the implementation didn't do it as a process, and now they typically just have more edge cases that won't delete cleanly.

What you missed from my original post is that it is a lot of effort to expose transitory state: nobody is going the extra mile to annoy you, they are just not investing in implementing extra UI to expose it.

And in most cases, it's the right balance.

From my experience, most places don't model the transition state. Such that it easily gets bunged.

If you just want to make it hidden, that is fine. But can lead to it's own problems. Usually when you have uniqueness constraints on things.

I think billions have been spent bridging the gap between “ideal” software and what businesses actually need. Access control is another thing I see developers wanting to simplify or push to implement later, but is actually a key feature.

There's definitely perfect and perfect for the business. But when large companies have many developers they all have to do something. They will spend their time doing something unnecessary.

Nowadays, regulations goes both ways. Data sovereignty regulations and GDPR-like laws can mandate that data must get hard-deleted.