The problem with an RDBMS is that the schema is retroactive -- every time you update the schema, both old and new data must validate against it.

There's another approach, which is to also version the schema. Every record is conceptually a pair of [schema, data]. This puts a burden on the application -- every client must be able to understand old schemas -- but the benefits are considerable.

The most trivial benefit is that no data is ever lost, even if the business case is taken away. Over time, the burden of supporting obsolete features mean that you want to remove columns that don't apply anymore. But that does remove interesting and potentially valuable historical data. If the schema evolves together with the data, you can keep even the cruft, at little expense.

In practice, the evolution of the schema means that data often has to be transformed from one schema to another (e.g. v1 -> v2). In an RDBMS, this transformation (schema migration) is performed once, and if it contains bugs, you have to resort to backups to restore the old data. But with a versioned schema, the transformations are simply functions on immutable data. You can go back and re-run transformations on old data to get new data, non-destructively.

CouchDB is cool if you want to mess with revisions: http://docs.couchdb.org/en/2.1.0/intro/api.html

When you're talking billions - trillions of events, check out something like https://www.slideshare.net/AmazonWebServices/how-netflix-use...

id uuid ts timestsmp data jsonb

solved it for you :) and thats will have ACID and all the other features of RDBMS that kafka does not have. It will also allow you to create transactionally consistent "views" representing whatever you need implemented in a few lines of SQL vs complex standalone services.

Not quite. The JSON overhead can become quite problematic. Not to mention, Postgres's JSON support isn't where I'd like it to be. :(

What features are you missing?

I think this is what Valve does with Dota 2 replays by using protobuf

Postgres could in fact be used here by creating an append-only table similar to:

id, data (JSON field)

However, Postgres doesn't have good support for creating derived append-only logs (=streams) from that table. Kafka has Kafka Streams and KafkaSQL. And producer+consumer APIs that are a good fit for NYTs use case.

> Am I missing something?

Remember that in addition to schema changes, NYT also wants to avoid row changes. One reason was that all search indices & other systems need updating too during a row change in a DB and this will lead to inconsistencies in large scale (sometimes some of these updates fail).

IMHO the thing you are missing is the log based architecture where all databases are derived/materialised from the SSOT: the log.

Pretty much all databases are already a transaction log that gets materialized. Postgres and many other RDBMS take in changes, write to the WAL, then provide tables that are views of the latest data of each row.

If you want total history and the database doesn't support this automatically then you can easily insert news rows instead (like you described) and then just create an SQL view that then shows the latest versions of each row, while also adding views for all kinds of other data access patterns. Companies have been doing this for decades because it's self-contained, fast, and reliable.

Using Kafka and separate processes to do this is deconstructing the RDBMS into separate layers that you now have to manage yourself. Useful if you really have that kind of scale but at 100GB of data, it's just silly. Use kafka as a work queue but leave the database work to actual database software.

However, Postgres doesn't have good support for creating derived append-only logs ???? A one line trigger will give you derived append only log that is transactionally consistent.

so how does that one-liner transform data from log A to log B in realtime?

are talking about creating a new table or materialised views or stored procedures or...?

In your post you are proposing to have a table be equivalent to a kafka topic. So derived "log"/topic can be another table updated with a trigger.

I agree.

It's rare that a company needs something like Kafka.

Kafka introduces a number of issues related to the development of client code and data stores (if any) and the maintenance of these things. It's important that the actual scale justifies the expense incurred.

It's not rare, using Kafka in place of a prober db is crazy though. There are tones of use cases that benefit from having Kafka a very common one is putting Kafka in front of the data processing pipeline to absorb spikes without data loss.

It's rare that a company has spikes of that sort that can't be handled in far more mundane ways. Most companies do not operate at a scale that merits this sort of solution.

Look at the engineering blogs on LinkedIn for the reasoning behind creating it. It's over-kill for most use cases.

Kafka is fairly mundane piece of software aside from periodic ops issues with Zookeeper.

Kafka doesn't have a schema per message, messages are just bytes that you can serialize / deserialize however you want to. The article refers to using protobuf for messages, which does support adding fields.

If you're equating kafka topics with the idea of a schema, you can add topics.

To add to this. Having implemented a similar log-based architecture, I would say that it is much simpler data infrastructure than having a central RDBMS considering the use case. Remember we need to deal with several consumer applications with their own respective optimization models. Postgres is a perfect choice for a given application's local datastore, while for a different application they may want to use ElasticSearch as their datastore. However, the "source of truth" remains free of any such optimization requirements. You simply save your "messages", "event", "facts", whatever you want to call it in its purest form preferably immutable, and let the consumer apps create/recreate their local datastores as they deem fit.

I tend to be the one arguing this, to stick to postgres for most things but even I will admit it does depend on scale.

I'm not sure what the NYT requirements are but from my understanding of Kafka, its persistent redundant distributed queues scale automatically horizontally across machines to support colossal amounts of data. It's possible that they had difficulty fitting everything in a postgres instance.

Kafka by default is not persistent, the logs expire after 7 days. You can increase it on a per topic basis. It also doesn't scale automatically. If you have three replicas on a single partition topic, they will live on their assigned broker forever unless you manually reassign them. Adding new nodes does not kick off rebalancing of partitions. Its automatic cluster management is very primitive compared to something like elasticsearch.

For example, if you lose a broker, the replica will just be gone forever. Unless you replace the broker with the same id.

Your statements about data retention are true, but if I may, I think they are only vacuously true. There are definitely ways to configure data retention incorrectly, but it seems like the folks at NYT have cracked the nut. Indeed, this is a trivial thing to set up.

Your points about auto-scaling are well-taken, though. It does not automatically rebalance when you add a new node. Confluent obvious agrees that this is a problem, since this is part of what Confluent Enterprise does. #shamelessplug (Which of course I do not intend shamelessly; just saying you have a point. :) )

See, that's where I'm confused. I'm no Kafka expert, but they say they use a "single-partition topic" which I believe means the only way they can replicate the data is by replicating the entire log, they can't shard because it's a single partition. The reasoning behind this is because Kafka doesn't support ordering between partitions.

Also I've never thought of Kafka as a persistent data storage solution, it's interesting Confluent is supporting Kafka being used in this way.

You are not even a little bit confused! I think you have it perfectly right. And you are not alone in not thinking of Kafka as persistent storage, but when you get down to it there is no reason not to, and people are indeed using it in just that way. And yes, Confluent does give its +1 to this practice. :)

>it's interesting Confluent is supporting Kafka being used in this way.

If it earns them money, I think they'll pretty much support anything. Jay has rubber stamped it on SO [1], but he's got a bit of a vested interest on selling Kafka.

1. https://stackoverflow.com/questions/17708489/using-kafka-as-...

The author states it's about 100GB of data.

Is 100GB of data really enough to warrant this type of architecture?

if you read more closely, this article is not about the amount of data, but the way how it is managed.

Precisely. While "all NYT data since 1851" sounds like a lot, and >8660 days sounds like a long retention period for a Kafka topic, this--like most systems in the world--is not a Big Data application. One of the key insights from the post is that there are interesting architectural considerations that have nothing to do with data size that make immutable logs a good idea.

They are storing 100GB :). You can do ok with 5TB +/- on something as crappy as RDS.

Heh this was exactly my thought. I've no idea why they couldn't just store the data in a database! Or if they really have several ways of storing data, then have a database for each and a common API/ESB layer over the top. Seems normal.

Postgres doesn't help with the streaming part though, does it?