Re:Publishing with Apache Kafka at The New York Times
by seanp2k2 on Thursday September 07, 2017 @05:14AM
(#15190170)
CouchDB is cool if you want to mess with revisions: http://docs.couchdb.org/en/2.1.0/intro/api.html
When you're talking billions - trillions of events, check out something like https://www.slideshare.net/AmazonWebServices/how-netflix-use...
Re:Publishing with Apache Kafka at The New York Times
by qaq on Thursday September 07, 2017 @06:52AM
(#15190589)
id uuid
ts timestsmp
data jsonb
solved it for you :) and thats will have ACID and all the other features of RDBMS that kafka does not have. It will also allow you to create transactionally consistent "views" representing whatever you need implemented in a few lines of SQL vs complex standalone services.
Re:Publishing with Apache Kafka at The New York Times
by manish_gill on Tuesday September 12, 2017 @06:01AM
(#15226034)
Not quite. The JSON overhead can become quite problematic. Not to mention, Postgres's JSON support isn't where I'd like it to be. :(
Re:Publishing with Apache Kafka at The New York Times
by qaq on Thursday September 14, 2017 @02:01PM
(#15250699)
What features are you missing?
Re:Publishing with Apache Kafka at The New York Times
by turbohedgehog on Thursday September 07, 2017 @02:03AM
(#15189456)
I think this is what Valve does with Dota 2 replays by using protobuf
There's another approach, which is to also version the schema. Every record is conceptually a pair of [schema, data]. This puts a burden on the application -- every client must be able to understand old schemas -- but the benefits are considerable.
The most trivial benefit is that no data is ever lost, even if the business case is taken away. Over time, the burden of supporting obsolete features mean that you want to remove columns that don't apply anymore. But that does remove interesting and potentially valuable historical data. If the schema evolves together with the data, you can keep even the cruft, at little expense.
In practice, the evolution of the schema means that data often has to be transformed from one schema to another (e.g. v1 -> v2). In an RDBMS, this transformation (schema migration) is performed once, and if it contains bugs, you have to resort to backups to restore the old data. But with a versioned schema, the transformations are simply functions on immutable data. You can go back and re-run transformations on old data to get new data, non-destructively.