News for hackers. Stuff that matters.
Re:Publishing with Apache Kafka at The New York Times by lobster_johnson on Wednesday September 06, 2017 @07:13PM (#15187895)
The problem with an RDBMS is that the schema is retroactive -- every time you update the schema, both old and new data must validate against it.

There's another approach, which is to also version the schema. Every record is conceptually a pair of [schema, data]. This puts a burden on the application -- every client must be able to understand old schemas -- but the benefits are considerable.

The most trivial benefit is that no data is ever lost, even if the business case is taken away. Over time, the burden of supporting obsolete features mean that you want to remove columns that don't apply anymore. But that does remove interesting and potentially valuable historical data. If the schema evolves together with the data, you can keep even the cruft, at little expense.

In practice, the evolution of the schema means that data often has to be transformed from one schema to another (e.g. v1 -> v2). In an RDBMS, this transformation (schema migration) is performed once, and if it contains bugs, you have to resort to backups to restore the old data. But with a versioned schema, the transformations are simply functions on immutable data. You can go back and re-run transformations on old data to get new data, non-destructively.

Re:Publishing with Apache Kafka at The New York Times by seanp2k2 on Thursday September 07, 2017 @05:14AM (#15190170)
CouchDB is cool if you want to mess with revisions: http://docs.couchdb.org/en/2.1.0/intro/api.html

When you're talking billions - trillions of events, check out something like https://www.slideshare.net/AmazonWebServices/how-netflix-use...

Re:Publishing with Apache Kafka at The New York Times by qaq on Thursday September 07, 2017 @06:52AM (#15190589)
id uuid ts timestsmp data jsonb

solved it for you :) and thats will have ACID and all the other features of RDBMS that kafka does not have. It will also allow you to create transactionally consistent "views" representing whatever you need implemented in a few lines of SQL vs complex standalone services.

Re:Publishing with Apache Kafka at The New York Times by manish_gill on Tuesday September 12, 2017 @06:01AM (#15226034)
Not quite. The JSON overhead can become quite problematic. Not to mention, Postgres's JSON support isn't where I'd like it to be. :(
Re:Publishing with Apache Kafka at The New York Times by qaq on Thursday September 14, 2017 @02:01PM (#15250699)
What features are you missing?
Re:Publishing with Apache Kafka at The New York Times by turbohedgehog on Thursday September 07, 2017 @02:03AM (#15189456)
I think this is what Valve does with Dota 2 replays by using protobuf