That is to say: as flexible as XML or an RDBMS schema with long-term, format encoded, data that can explicitly support conflicting clients over time (as desired by the dev). Zero-impact, live, online, updates touching hundreds of active systems without issue...
TBH most posters here have completely missed the forest for the trees. The point is not to avoid DB migrations. The point is to support hundreds of DB migrations in connected systems simultaneously with no schema-related down-time or centralized point of failure or intractable CAP challenges.
Trying to solve these kind of operation issues with an RDBMS in an Enterprise context is _literally_ the "big ball of mud" design pattern.
Kafka, warts and all, is an operational answer to how 1 client can feed 1 MM real-time connections, how massive resource unlimited batch systems can integrate with real-time feeds, how your data warehouse can keep growing without painful forced restructuring, and how data architects can mandate standards across multiple systems built by external teams with human sized budgets.
Data is part of it. Protocol, format guarantees, and loosely coupled systems are where the wins lie.
Sure but backwards and bi-directional compatibility inhibits the evolvability of schemas. Something as simple as adding a new required field for example is not backward compatible in avro.
I understand why this is, and that in a huge database, it's not that simple in SQL either. But in 95% of databases it actually is quite simple.
Seriously: that would be a show stopper beyond show stoppers, and would make Kafka useless. Backwards and bi-directional compatibility are the specific facilities that enable safe schema evolution.
From the docs:
"Backward compatibility means that data encoded with an older schema can be read with a newer schema... Forward compatibility means that data encoded with a newer schema can be read with an older schema... Full compatibility means schemas are backward and forward compatible[:]...we can evolve the schemas in a fully compatible way: old data can be read with the new schema, and new data can also be read with the old schema."
A new required field _with a default value_ (important!), is backwards compatible. Same deal for hardcoded SQL updates: no default value, no backwards compatibility. Default value? Backwards compatible because clients that cannot provide a value get one, by default.
And, frankly, if you're thinking about database in the singular form then you're not thinking about an environment that really needs a schema evolution policy... tens and hundreds of parallel databases, active datawarehouse operations, multiple federated service gateways, real-time analytics, and huge batch systems for BigData, all acting in concert 24/7: that's the kind of environment that makes the "95%" simple jobs massively costly due to the logarithmic interaction of complexity when numerous systems are considered at once.
Without some kind of dataflow separation these intrinsic complexities trend towards intractable system configurations, hence the ESB boom and bust. Kafka is not for your applications. Kafka is for your Enterprise with many, many, applications.
In what sense is this field required when you attempt to produce a value using the new schema, and everything is fine if you omit the "required" field?
> And, frankly, if you're thinking about database in the singular form then you're not thinking about an environment that really needs a schema evolution policy
We don't have 100s but certainly 10s of databases. But we use a schema evolution policy because we're attempting to use kafka as the system of record for our event log.
(1) Protobufs are not self-describing. There is a meta-model but you can't really understand the data unless you have access to the schema. This means just to be safe you might end up storing the schema beside every message or at least a reference to the schema. Welcome to the schema management business.
(2) Protobufs aren't extensible. In practice this means any time a system wants to introduce a new field they have to submit a proposal to some sort of highly centralized "Architecture Group." This is followed by lots of debate. Then, if you're lucky, it gets put in and a few systems adopt it. With XML you can slice off namespaces and let people innovate in those namespaces.
(3) Protobufs aren't human readable. At the end of the day this means you need special tools to do anything with them. Meanwhile XML can actually be imported directly into Excel.
There's a whole ecosystem of powerful technologies around XML that make it work very well in this case. People underestimate how valuable this is because they're still not thinking of the log as its own first-class product.
That said, perfect is the enemy of good. This could be a good step in the right direction and migrating to XML in the future would be pretty easy because it becomes a very simple event transformation.