Re:KSQL: Open Source Streaming SQL for Apache Kafka
by tedmiston on Monday August 28, 2017 @09:47PM
(#15121559)
It sounds good in theory. In practice, I haven't heard much around running backwards migrations on a data warehouse / massive collection of events but I'm sure some out there already do it.
Re:KSQL: Open Source Streaming SQL for Apache Kafka
by dkersten on Tuesday August 29, 2017 @11:06AM
(#15124772)
I suppose one needs to take care that migrations are never lossy so that the full information for upgrading or downgrading a version is available.
Re:KSQL: Open Source Streaming SQL for Apache Kafka
by tedmiston on Tuesday August 29, 2017 @11:18AM
(#15124847)
Yeah, that's the challenge. For instance, how do you handle when a column was one data type but then down the road was changed to another type when the two aren't cross compatible or could potentially break?
Re:KSQL: Open Source Streaming SQL for Apache Kafka
by raulk on Wednesday August 30, 2017 @10:35AM
(#15132421)
You could retain this info in a meta field of flexible type. For a DB, it could a JSON type. For messages, it could be an extra _meta field on the message that the systems themselves ignore.
Then if you have 90% confidence you'll only ever need to replay the upgraded stream, you can upgrade it and destroy the previous version.
If at some point (the remaining 10%) you need to rescue the old stream, you can run the "down" direction and rehydrate the old version of the stream.