News for hackers. Stuff that matters.
Re:KSQL: Open Source Streaming SQL for Apache Kafka by stingraycharles on Monday August 28, 2017 @02:59PM (#15119122)
As always in these types of the scenarios, the answer is: it depends. It depends on the amount of data you have. It depends upon how big the diversion from the original schema is. Etcetera.

My personal philosophy is to always leave event data at rest alone: data is immutable, you don't convert it, and you treat it like a historical artifact. You version each event, but never convert it into a new version in the actual event store. Any version upgrades that should be applied are done when the event is read; this requires automated procedures to convery any event version N to another version N + 1, but having these kind of procedures in place is good practice anyway. Some might argue that doing this every time an event is read is a waste of CPU cycles, but in my experience this far outweights possible downsides of losing the actual event stored at that time in the past, and this type of data is accessed far less frequently than new event data.

Re:KSQL: Open Source Streaming SQL for Apache Kafka by dkersten on Monday August 28, 2017 @04:01PM (#15119594)
I suppose you can always trade those CPU cycles off against storage and cache the N+1 version (in a separate Kafka topic or elsewhere), so now reading the latest-version data is fast, yet you still retain the original data intact, at the expense of more storage. This does complicate the storage though, as you now have multiple days a sources, but nothing that can't be solved.
Re:KSQL: Open Source Streaming SQL for Apache Kafka by raulk on Monday August 28, 2017 @05:02PM (#15120036)
Bingo. If you equate this technique to a DB migration, you could have "up" and "down" directions for the translations from version N <=> N+1.

Then if you have 90% confidence you'll only ever need to replay the upgraded stream, you can upgrade it and destroy the previous version.

If at some point (the remaining 10%) you need to rescue the old stream, you can run the "down" direction and rehydrate the old version of the stream.

Re:KSQL: Open Source Streaming SQL for Apache Kafka by tedmiston on Monday August 28, 2017 @09:47PM (#15121559)
It sounds good in theory. In practice, I haven't heard much around running backwards migrations on a data warehouse / massive collection of events but I'm sure some out there already do it.
Re:KSQL: Open Source Streaming SQL for Apache Kafka by dkersten on Tuesday August 29, 2017 @11:06AM (#15124772)
I suppose one needs to take care that migrations are never lossy so that the full information for upgrading or downgrading a version is available.
Re:KSQL: Open Source Streaming SQL for Apache Kafka by tedmiston on Tuesday August 29, 2017 @11:18AM (#15124847)
Yeah, that's the challenge. For instance, how do you handle when a column was one data type but then down the road was changed to another type when the two aren't cross compatible or could potentially break?
Re:KSQL: Open Source Streaming SQL for Apache Kafka by raulk on Wednesday August 30, 2017 @10:35AM (#15132421)
You could retain this info in a meta field of flexible type. For a DB, it could a JSON type. For messages, it could be an extra _meta field on the message that the systems themselves ignore.
Re:KSQL: Open Source Streaming SQL for Apache Kafka by vog on Tuesday August 29, 2017 @06:02AM (#15123224)
> this requires automated procedures to convery any event version N to another version N + 1, but having these kind of procedures in place is good practice anyway

Note that this "good practice" already has a name, it is usually called "migrations".

Migrations may be as simple as SQL Update / ALTER TABLE statements. But they may also be transformation of JSON/XML/... structures, or any other complex calculation.

It may not be the best term, as "migrations" usually imply that the result is written back to the data store. But apart from that, I don't see any problem using this term here.

Re:KSQL: Open Source Streaming SQL for Apache Kafka by daddykotex on Monday August 28, 2017 @03:03PM (#15119146)
Ok this makes sense.

It matches what some others have been saying as well. Thanks

Re:KSQL: Open Source Streaming SQL for Apache Kafka by stingraycharles on Monday August 28, 2017 @03:15PM (#15119237)
There's a whole world out there about this kind of stuff. Take a look at CQRS and some of the posts by Greg Young; they're highly informative and one of the first people to really capture this way of dealing with data properly.
Re:KSQL: Open Source Streaming SQL for Apache Kafka by _pmf_ on Tuesday August 29, 2017 @06:19AM (#15123295)
I wonder what happened to Greg Young. I think he had a book in the pipeline (which I was looking forward to), but as far as I can glean from social media, some burnout related stuff happened.