Have you considered using some other tool there? I know there is a Kubernetes target for it. Of course, any non-GCE port of Kubernetes is in likely need of sponsorship/love. And of course, Kubernetes currently introduces some additional network overhead.
Still, it might be helpful there for handling problems like getting persistent storage mixed in.
If you're interested in getting involved in Kubernetes-on-something-else, we'd love to hear from you. Drop by Slack at slack.k8s.io!
Can I run more than 1 cluster per account AZ without a silent but very nasty failure condition where both clusters become wedged?
I have stuff in production on it and I need to take it down because the entire cluster is unreliable. Mostly because there are frustrating bugs in disk space management that hurt clusters.
I really like Kubernetes and you can see I've carried water for it here. I've been trying to tie it to mbrace because thatz be a nice cloud portability. But I'm much more impressed with the GCE experience than the AWS experience.
https://github.com/mesos/kafka
Mesos supports persistent volumes and what they call "dynamic reservations", which this framework supports. If the application is killed, and the host is still up, it will be restarted on the same host and re-attached to the volume which has the data, otherwise, kafka will just replicate to the new broker (as it does).
The initial code for adding the dynamic reservation bits was added last November as it is a relatively new feature in Mesos:
Caveat: We only use it to run state-less applications. Applications like Kafka that require data persistence are run outside Mesos as data persistance is not yet handled reliably with Marathon.
[1] http://datajet.io/One-year-with-Apache-Mesos-The-Good-The-Ba...