News for hackers. Stuff that matters.
Re:Data denormalization is broken by swlkr on Friday September 30, 2016 @01:19PM (#12613773)
I can see why the normalized approach is difficult to make performant, but it seems much simpler.
Re:Data denormalization is broken by arjunnarayan on Friday September 30, 2016 @02:22PM (#12614226)
You're correct on both counts. A normalized approach is simpler, and is more difficult to make performant. So you start there, but as you scale, it becomes more and more untenable for your users as your latency goes up the wazoo.

Thus, eventually your app hits a scale where you have to rewrite your code to explicitly denormalize things and maintain the denormalized views yourself. This makes any feature addition/backend update horrendously complicated, and everyone's life difficult. You slowly start losing consistency which causes "glitches in the matrix" for your users. To stop that you start adding in transactional semantics yourself at the application level, checking and cross-checking everything, and doing abort/restarts, and then finally, in an analogue to Greenspun's tenth rule[1], your application starts to contain an ad hoc, informally-specified, bug-ridden, slow implementation of a distributed database.

I think there's a CircleCI blog post parody hidden inside this comment, but I'm not clever enough to materialize it[2].

[1]: Any sufficiently complicated C or Fortran program contains an ad hoc, informally-specified, bug-ridden, slow implementation of half of Common Lisp., https://en.wikipedia.org/wiki/Greenspun%27s_tenth_rule

[2]: A reference to https://circleci.com/blog/its-the-future/

Re:Data denormalization is broken by jacques_chester on Friday September 30, 2016 @03:40PM (#12614755)
Broadly: Normalised data is more performant on write, because for each piece of state, there will be one and only one place to store it.

It's less performant on read because you are constructing logical deductions out of a lattice of tiny pebbles of truth. It takes time to find, filter, sort and project those.

Denormalisation "works" by skipping to a read-optimised schema for those applications where reads greatly outnumber writes. Which is very many.

As an aside, "denormalised" doesn't have to mean "unstructured chaos". Do As Thou Wilt isn't necessarily the law of denormalising. There are well-studied, widely-used denormalisation techniques under the heading of "dimensional modelling".

It's just that most of these discussions are dominated by web folk, not data warehouse folk.

The shoutout to Event Sourcing is promising, given the interaction with the CQRS pattern. In short: read schema and write schema have different demand and performance characteristics. So split them up.

Re:Data denormalization is broken by LgWoodenBadger on Friday September 30, 2016 @04:00PM (#12614893)
I don't know that you can always conclude that normalized data is more performant for writes.

Take for example the case where your normalized write consists of 5 insert statements into 5 tables, instead of 1 insert into 1 table that you'd do with a denormalized write.

Re:Data denormalization is broken by jacques_chester on Friday September 30, 2016 @04:42PM (#12615164)
As always, consult your doctor, accountant, lawyer and profiler.