erichwang

Making the schema management simpler was definitely one of the things we wanted to do here. We wanted it to be easy to set up and maintain, and schema inconsistency is one of those things that has been a continued sore spot from my past experiences. The only caveat here is that our post only discusses data that is already in Kafka for the most part. To have a complete solution, ideally the client submitting the structured event to Kafka would also recognize and enforce that same serialization schema -- which is a bit of harder problem to solve in a general sense, but there are ways to do this.

Yep, this came out from trying to figure out how to set up something that would be generally simple to set up and maintain, but still be scalable for future growth. There didn't seem to be a lot of examples, and so we thought we would share this and give the exact instructions on how to get something like this running.

This is more of a definitional thing. The default parameters provided in the blog example provide roughly a 60s time-to-query latency. For most folks, this is probably effectively real-time, but for others it might be too slow for other use cases. That being said, this time corresponds to the underlying Flink flush interval and this value can easily be tuned to whatever value you want it to be. For example, you could make it 10s if you wanted. However, if you do reduce the flush latency, you may want to increase your compaction frequency to balance that out.