Something the original post reminds me of a little is Crux [1].
[1] https://www.confluent.io/blog/turning-the-database-inside-ou...
(Nice to see it's still relevant – the article is linked at the very top of the crux about page: https://opencrux.com/about/what-is-crux.html)
The end result you speak of is a more flexible data platform, which can handle more use cases, including applying decoupled parts to different things.
A good example, in my mind, is Apache Spark - which is like taking a compute and SQL query engine, making it distributed and decoupling it from the underlying storage. Now, suddenly SQL or code can be used on anything you can fit into a data frame, even streams - wow!
Pulling out the transaction log - and wow, suddenly we have durable, potentially immutable queues we can use for even long term storage.
I was so excited by the line of thinking in that article. :)
Words have fuzzy boundaries - immutable obviously doesn't mean actually immutable. It means immutable like how git is immutable - you're not meant to change history. It'll fight like a tiger to stop you easily changing history. It'll give you two different ways to get what you want while preserving history. But if you want to change history you can.
SQL is said to be declarative but if you get hit dealing with the wrong sort of workload in SQL then the implementation details can start to matter extremely quickly.
In practice, the author probably only means 'much more immutable' rather than 'illegal to use with European data' level immutability.
I ran pretty important systems as a DBA years ago that we’re significant but would never, ever require a meaningful constraint on growth. (Performance, different story)
Sometimes looking at a problem like a database with a new set of constraints leads to interesting and novel things.
Professionaly I work on a financial reporting system where there are around 1.5 million accounts with trading activity. Things like moving a month of data around and checksumming still take on the order of 1 hour. If the size of that monthly dataset doubled, we'd be OKAY but it it grew 10x we'd have to start restructuring things both on the software side and the hardware side.
For personal work I am working with searching video frame substrings regardless of codec, resolution etc within videos in a ~50 TiB video set. Having to re-encode hundrends of GiB of video at a time is some thing that takes a huge amount of time and nothing something I can just "throw into an S3 bucket". The growth of said working set is not truly constrained but is actually constrained. I can only afford so many 8-10 TiB HDDs at a time! They also fail and have to be mirrored/backedup with rsycn at the very least which adds to the cost for me.
Think about what you’re saying about your personal project. What you’re talking about wasn’t financially achievable for TV companies just 20 years ago. Hell, anything older than a few days would be stored offline in some tape silo, or in a shelf. (And 10-12 people to attend to it!) Now you as an individual could afford to do all sorts of stuff with that video data, albeit for some significant $.
unsure whether to laugh or cry
My solution was to provide an undo-chain of a fixed given length. Versions older than that simply "fall off the chain" and will be picked up by GC eventually. You can also make snapshots long-lived snapshots if you want.
(Of course removing a snapshot is an append-only operation :P but GC will go after non-referenced snapshots eventually.)
Darcs is different from git in that a "commit" (called a "patch" in Darcs lingo) does not depend upon the entire history of what has come before. Rather it has specified dependencies, which Darcs can infer itself – or you can specify if you want more semantic content in the patch structure.
A repository is a collection of patches, which are merged using something called patch algebra. So, if you want to delete a patch, you just issue the command "darcs obliterate" and it will remove the patch, and anything which depends on it. Then working directory is reconstructed by merging together any loose ends using patch algebra.
With careful administration of dependencies, this approach makes deleting specific things very slick.
It means you separate two use cases: retracting information (I remove an item from my shopping cart because I want another item) and purging data (I delete my account and all my shopping history because I don't want the shop to store my data anymore).
You can still purge data easily in Datomic, for example.
Most of GDPR is not really new. There was a significant amount of teaching about data protection when I was at university in 1994-1997 and it wasn’t even new then.
That is so backwards I can't take the guy seriously.