Publishing with Apache Kafka at The New York Times

Re:Publishing with Apache Kafka at The New York Times by amenod on Wednesday September 06, 2017 @02:28PM (#15185951)

> Kafka messages are immutable. Each of those green boxes on the right hand side of the first diagram will need to have special-case logic to unpack the kafka stream, with knowledge of its changes (up until 17 May 2017, treat the data like this, but between then and 19 May 2017 do x, and after that do y).

I respectfully disagree. The genius of this approach is that you can make the same transformation on the original Kafka stream to change its schema and prepare the new feed. Once you are satisfied with the results and you have switched all subscribers to the new feed, just turn off the old one. Voila - you only have y.

> This is a rare case where use of XML makes sense.

Sorry, but no. Just no.

[ Parent | on HN ]

Re:Publishing with Apache Kafka at The New York Times by cturner on Wednesday September 06, 2017 @03:50PM (#15186591)

"Voila - you only have y."

Thanks. My mental model had a long-lived channel, but I follow your explanation.

[ Parent | on HN ]

Re:Publishing with Apache Kafka at The New York Times by btgeekboy on Thursday September 07, 2017 @12:16AM (#15189102)

I still don't understand the hatred around XML. Is it slightly verbose? Yes. Does it support lots of neat functionality that make it great for interoperating between systems, like validations and transformations? Yep. Sure, it's possible to go full architecture astronaut with it, but you can do that with pretty much any programming language.

Meanwhile, I'm just sitting over here wondering whether my YAML file is supposed to have certain indents here or not, "-" or not, and trying to go figure out which magic incantation I need to get it to handle a multi-line string the way I'm expecting.

[ Parent | on HN ]

Re:Publishing with Apache Kafka at The New York Times by annnnd on Thursday September 07, 2017 @02:31AM (#15189543)

I think parent was answering the usage of XML in this use case, which is not appropriate. XML has many strengths (as you have outlined), but it has also been (mis/ab)used so many times that it gained bad reputation. What my GGP suggests is an example of such. There is nothing to gain from XML here that any proper DB (with schema) wouldn't offer, or in this case, protobuf.

Kafka logs however are solving a different problem. The mental model is different - they do not record state, but the whole history of transactions, which makes it trivial to change the schema if/when need arises. Saying that the schema should be thought in advance and shouldn't change is not realistic IMHO.

[ Parent | on HN ]

Re:Publishing with Apache Kafka at The New York Times by wolfgang42 on Thursday September 07, 2017 @10:46AM (#15192296)

> trying to go figure out which magic incantation I need to get it to handle a multi-line string the way I'm expecting.

Shameless plug: https://yaml-multiline.info is a little website I wrote to help with exactly this. I'd love to hear what other people think of it!

[ Parent | on HN ]

Re:Publishing with Apache Kafka at The New York Times by robertlagrant on Wednesday September 06, 2017 @05:18PM (#15187210)

Not sure how just having a new API on top of a database doesn't achieve the same?

[ Parent | on HN ]