News for hackers. Stuff that matters.
Re:Immutability, MVCC, and garbage collection by RyanZAG on Sunday December 29, 2013 @12:21AM (#6978521)
What? Come on, really? My database - only the latest version of all the data that gets updated frequently - has TB of data. My whole git repo with full history is about 500mb and that's excluding large assets. It's a completely different ball game. Sure, if I could buy 100TB disks cheaply and set it up so I could access data across hundreds of them in real time then there wouldn't be a problem. Obviously, I can't do that.

As the author of the blog post points out: the little thing called reality gets in the way of such nice and perfect arguments and ideas.

Re:Immutability, MVCC, and garbage collection by jasonwatkinspdx on Sunday December 29, 2013 @04:14AM (#6978966)
When talking about architecture at this level, you need to take a longer view. Once upon a time, a gigabyte database was mammoth beyond thought. Also, you're taking it as given that your data has its present storage footprint even though alternative architectures might have very different properties. It's worth noting that a purely additive database can be very aggressive about compression.

But also, petabyte datasets are downright cheap to deal with these days. They'll look more like that half gigabyte source code repo sooner than I think you're considering.

Re:Immutability, MVCC, and garbage collection by RyanZAG on Sunday December 29, 2013 @07:36AM (#6979387)
We're obviously coming from different backgrounds here - do you work at Google or similar ?

'Downright cheap' is a Heroku instance or two. 'Affordable' is 20 or so servers with a few TB of drive space each in a cluster with a few backups. Petabytes of data needs a huge number of servers and dedicated personal to look after - I don't know how that can be regarded as 'downright cheap'.

If you're in a position to handle petabytes of data you're probably going to roll your own data center with custom software ala Google's bigtable. This definitely isn't the market that these little startups are targeting as far as I can see. The OPs critique is very well founded.

Re:Immutability, MVCC, and garbage collection by corresation on Sunday December 29, 2013 @12:16PM (#6980257)
petabyte datasets are downright cheap to deal with these days

Are you from the future? In no universe is a petabyte database remotely close to being cheap, even for large organizations. At this point I am seriously questioning your industry knowledge.

Re:Immutability, MVCC, and garbage collection by empthought on Sunday December 29, 2013 @01:49PM (#6980684)
Sure it is, a few thousand entries pointing to 10-100GB pirated HD video blobs. What's the problem?

/s

Re:Immutability, MVCC, and garbage collection by thirsteh on Sunday December 29, 2013 @01:17AM (#6978640)
You're right, but the vast majority of databases aren't that big. You are very explicitly recommended against using something like Datomic if you think you will need to purge data with regular intervals.
Re:Immutability, MVCC, and garbage collection by mgkimsal on Sunday December 29, 2013 @08:53AM (#6979513)
"but the vast majority of databases aren't that big" probably because they don't keep all copies of everything forever. Even as basic blog system would grow huge just based on spam comments alone.
Re:Immutability, MVCC, and garbage collection by thirsteh on Sunday December 29, 2013 @09:11AM (#6979552)
But you don't have to keep redundant copies around forever. Immutability means nothing in the past can change, which means it's perfectly fine for future entries to reference old values. New rows can essentially be deltas, much like version control systems.

I disagree that Datomic necessarily grows huge. I'm just saying don't use it if you know you're going to be storing terabytes and you don't have enough disk space to handle it without needing to truncate data with regular intervals.