If virtual machines with live migration is what you're after, vmware is not your only option. You can get largely the same effect without involving any iscsi/FC/SAN tech with another open source project, which happens to originate from the same place as Kubernetes: http://www.ganeti.org/

Ganeti has been battle-tested for about a decade too, supports an assortment of storage backends, including some clustered ones like Ceph, can do live migration between hypervisor nodes, and it's a nicely maintained Python app with some Haskell parts in it.

ganeti came out of my team at google in the mid 2000s, fwiw. I did not work on it, but I've certainly used it. It's pretty nice.

Thank you. I've used Ganeti extensively and it help wrangle our Xen/DRBD rats nest very quickly. It's an awesome project.

> I don't want to break your hopes but stateful containers will only ever run on GCE and AWS.

Actually Kubernetes is starting to work on support for persistent local volumes; we know the lack of this feature is a significant barrier for running some stateful applications on Kubernetes, particularly on bare metal. The concrete proposal for how we are thinking to do it is at https://github.com/kubernetes/kubernetes/pull/30044

The high-level feature requests are at https://github.com/kubernetes/features/issues/121 and https://github.com/kubernetes/kubernetes/issues/7562

(Disclosure: I work on Kubernetes at Google.)

Having containers with local volumes is counter productive. They're just pet that can't be moved around and killed/recreated whenever you want. (Though I understand that it can be useful at times for some testing).

IMO: It's a marketing and usage problems. You should re-focus people on running exclusively stateless containers. Sell the strengths of containers, what it's good at and what it's meant to do. Containers = stateless.

Stateful containers are an hyped aberration. People barely get stateless containers working but they want to do stateful.

I don't understand why you say "Having containers with local volumes is counter productive." I would agree it's probably not a good architecture if you're running a huge single-node Oracle database, but it's an excellent way to run data stores like Cassandra, MongoDB, ElasticSearch, Redis, etcd, Zookeeper, and so on. Many people are already doing this, and as one large-scale real-world example, all of Google's storage systems run in containers. The first containerized applications (both at Google and in the "real world") were indeed stateless, but there's nothing fundamental about containers that makes them fundamentally ill-suited for stateful applications.

You don't understand because you are blinded and spoiled by Google.

Go see the outside world => They have none of your internal tech and services. Stateful containers do not exist there. "Containers" means "docker" which is experimental at best.