Going to port to confluent-kafka-python this week now that we updated our cluster to Kafka 0.9. PS: you should try providing binary distributions (wheels), though. Having to build librdkafka + require GCC to pip install from source is not friendly. If there's any interest I can help on this (will have to do anyway).
I'm actually in favor of breaking with Zookeeper (It's terribly designed, has serious problems with concurrency and even consistency, and has the classic java problem of using java's internal serialization methods, which are utterly unfit for storing more than temporary data on a single host.) However, absorbing all of the demands of a lockservice into every product is not the solution to Zookeeper's failures.
This is important for three reasons: 1. To make clients easy to build 2. To have a single security layer for everything (so you don't have to authenticate yourself with two things) 3. To be able to enforce things like throttling and protect the system in an environment with lots of clients.
It's true that when we were working at LinkedIn Kafka tended to have much better Java support. Since founding Confluent (I'm one of the co-founders) we've really focused on improving the situation outside Java.
A few specific corrections:
1. We added full support for consumers with no interaction with zookeeper in the main kafka protocol. There is no longer any direct interaction with zookeeper from either the producer or consumer. We did this because we care a lot about the non-java clients.
2. Kafka has been extremely disciplined about backwards compatibility. The protocol comes with versioning and changes are always implemented in a way that supports both the old and new version and can be rolled out without downtime. In the five year history of the project we did one backwards incompatible release--the break from 0.5.x-0.7.x to 0.8.x. This was done intentionally to allow us to refactor the apis. I think this is a pretty good track record.
It's worth also addressing why Kafka clients directly access nodes in the cluster rather than requiring a proxy layer. The reason we do this is to allow very high throughput, partition aware processing. This is really required for use cases like stream processing that need to process data efficiently, especially in cases where you are reprocessing data. You can always build a proxy layer on top of direct access but not vice versa.
Confluent (where I work) is doing two things that help the non-java client ecosystem: 1. We maintain an open source REST proxy that provides decoupled access (albeit with a little overhead compared to the direct clients) 2. We have picked up work on clients. We offer and fully support a c/c++ client, a python client, and have a Go client coming soon. All of these are in feature parity with the Java clients. (More on the way).
Both of these efforts are open source and apache licensed and included in the open source Confluent Platform distribution of Kafka.