That being said, I believe it is using a vector clock, but I'm not sure.

Netflix probably saw an opportunity to built the Dynamo layer on top of what Twitter had already written, and decided to just keep building on in C instead of rewriting a fairly large codebase.

And that's the magic.

To boot, they do it at enormous scale on a multitude of platforms all while their own destructive code (Chaos Monkey) runs around destroying servers.

It's true, some things like "stream me this" are maybe not rocket science, but at Netflix scale, everything becomes a bit more difficult.

Their requirements are 100% availability, and an API's that never breaks, but consistency rarely matters.

The engineering effort is cool, but it is solving a problem that I don't have. The only thing that matters to netflix is they list the catalog somewhat correctly, and always deliver a stream.

I have a different problem, thus I am uninterested in their magic. If I had a similar problem or they signed my paycheck, I'd care.

http://netflix.github.io/#repo

Early on I remember the heroics that Netflix went through running on EC2, building numbers of complex tools for varied performance, reliability issues, etc. At the time many much larger sites just quietly worked 24/7, minus the heroics, and minus the effort, often by using purpose-suited dedicated servers. Later, once Amazon rolled out SSDs, Netflix triumphantly announced how much of an improvement it was to their product, again demonstrating that they were creating a problem (huge numbers of horrible I/O machines) that they then solved with gusto, albeit unnecessarily.

I'm not trying to be overly down on Netflix, but it is a company that seemed to make tech blog entries a product of the company years ago, and the result is that people have bought into this notion that they're doing some hugely complex task. They aren't.

Sorry, but this is an invention of yours. No one said they're doing anything flippantly.

But they may be doing it ignorantly: Success in one field (in this case turning a mail subscription business into a stream service) doesn't imply technical innovation or leadership, and often is despite it.

This industry is absolutely rife with people creating solutions to problems they themselves invented and caused. In the case of Netflix, an enormous amount of their solutions have been founded around the notion of deploying on huge numbers of miserable Amazon EC2 instances, and then dealing with the problems related to that. Others simply deployed distributed data centers hosting their own purpose suited, reliable hardware with big fat storage arrays, and the problem is solved. It's like trying to build a car out of toothpicks and then detailing the innovations you created in toothpick redundancy and robustness.

That just isn't the case in the real world when working with tens of millions of customers. This will become apparent when you actually get out there in the field and realize systems involving tens of thousands of machines hosting dozens of different services to millions of people is actually not simple.

There are a large number of services of the user scale of Netflix, with dramatically more complexity, that don't seem to have the heroic issues that Netflix does. Facebook and Reddit both are of a complexity scale multiple-magnitudes greater than Netflix. Someone else absurdly tried to draw Apple and Google in, but again, what they do absolutely dwarfs the entirety of Netflix's operation, to the point that it becomes almost laughable.

Netflix is a very simple application. Scaling it can be difficult, of course, but this is hardly some new grounds. Further, the particularly make-up of Netflix makes it one of the most profoundly scalable platforms going (silos, little to no need for transactional integrity. It is profoundly simple). You don't have to believe this, but your comically patronizing responses just sound...laughably unskilled.

Have you actually work with any of their backend engineers, seen any of their infrastructure and why they made the design decisions they did, or actually talked to anyone at Netflix, or is this all just speculation?

I'm sure they'd love to hire you to to greatly simplify their overly complicated architecture otherwise...

What may seem fairly low value from the outside or consumer perspective is likely of huge value internally to ensure you can always watch Netflix without even thinking about everything that has to happen to deliver as much video as they do to their customer's browsers.

What exactly is trivial about building a complicated multisource streaming video service with sophisticated recommendations (plus all the other "little" stuff like payments)?

Google is "trivial." It's just a box for doing text search on a graph database. Apple is "trivial." They just make pretty skins for commodity hardware & software. Etc.

When you consider 99% of everything in your field trivial, maybe you should consider refining your definition of trivial.

And if we're being honest, the Netflix app layer is kind of terrible. I could enumerate the problems, discoverability being king, but it is not the compelling part of the service.

Meanwhile, app level features like restricting certain types of content isn't available. I want to be able to delete crap from the Netflix catalog when my kids log in.. I don't care about nudity, language, etc. but more about just how dumb some of the cartoons are.

Also, I noticed comments about an architecture document here: https://github.com/Netflix/dynomite

Anyone know where the doc they are referencing is located?

However, since it is open source, contributions are welcome! :)

However, you can imagine different use cases for each of these stores and you can potentially increase operational efficiency by having a common layer which handles the distribution of data for each store.

DC := AWS region

Rack := AWS AZ

That should tell you that they need multi-region availability, something that DynamoDB will likely never provide.

https://github.com/Netflix/dynomite/blob/master/src/dyn_dnod...