Re:Importing JSON into Hadoop via Kafka
by vlahmot on Wednesday January 18, 2017 @08:30PM
(#13432181)
The schemas are not immutable. You also don't hit the schema registry for every message either, in fact you can skip the registry all together and provide the schema manually if you would like.
Re:Importing JSON into Hadoop via Kafka
by lacksconfidence on Wednesday January 18, 2017 @11:44PM
(#13433079)
You could provide them manually, but then any schema upgrade becomes a big pain. Wikimedia, as one example, uses versioned schemas. As such each version is immutable and can be pulled from the cache. Each kafka message has a null byte, and then a long version number prefixed to indicate how it should be decoded.
https://github.com/wikimedia/analytics-refinery-source/blob/...
Why would they need to hit the registry for every message? Wouldn't the schemas be immutable and thus able to be (at least temporarily) cached? They might have millions of messages but it's doubtful they have millions of message schemas.