I think it's all about creating ready to use containers which are running on different hosts which can replicate their data between themselves and handle failovers (they need to connect to etcd or zookeeper or consul for that).
For PostgreSQL it seem to have already solved (?) 2 times by different people:
https://github.com/zalando/patroni https://flynn.io/
But we don't only need it for PostgreSQL.
You need ready to use data-replicating Docker images for storing/indexing log files.
And for statistics.
Probably ready to use containers for replicated/clustered Redis.
We need something for a replicated S3-like service for storing static files.
Which can be used for different things like: - let's say you are running a website with Wordpress it can be configured to put it's files in a 'CDN'. So now whenever you you deploy Wordpress you point reverse proxy with caching (like Varnish) at your S3-like service.
A S3-like service could also be used to hold your own Docker images. You could put a Docker registry in front of it to push and pull to.
We need something, maybe also S3-type service, for storing the files of your local git repositories (which holds the sources of your Docker-images).
As far as I know in the ecosystem there is still a scheduler missing which can deploy these containers on the right hosts.
And there is also no standard API for starting new containers or whole machines when the other containers (or some monitoring tool) in a cluster noticed one is missing.
These things take time. Lots of time. :-(
Our Postgres appliance is unique in that it is explicitly designed to not lose data in the face of failure. Simply wrapping up a database in a container with leader election and replication is not enough, as there are many pitfalls that can cause data to be lost during failures. Of course we are limited by the guarantees the database can provide, so some datastore appliances we build may have clearly defined caveats.
Storing binary blobs is another tricky matter, and we're exploring what we can do with little to no configuration. We currently store app images and git repos in Postgres, which works but will only scale so far.
We are also improving our scheduler so that it is capable of providing the constraints necessary to place stateful services properly.
Anyway keep up the good work.
2. You can point Docker to a file - use 'docker save' and 'docker load'.
Sometimes it gets hard to follow everything in the Docker ecosystem.
Yep - and the ability to run just one per node is coming "soon" (ideally by next milestone). It's something we wanted to get in, but just didn't have the time by 1.1.
My #1 recommendation is to do whatever you do today. If you mount an NFS mount into your VM to store your MySQL data, that's what you should do again. The speed at restarting a pod is very low - but you do need to do the (small) hack today if you want to be absolutely sure you don't have multiple pods with on the same node.
To be clear, the ugly-ness is one line in your config file:
nodePort: <some arbitrary number >1024 & <65536>
You could indeed create seperate replication-controllers against labels, such as your aws-zone, and run them seperatly. This flies in the face of having replication-controllers and sucks in my opinion.
The other option is a single replication-controller and have your containers use a resouce that can't be reused, thus ensuring you're scheduled elsewhere. This is hacky.
I suspect it'll be another year before kubernetes is all singing and dancing, but I do like using it.
Edit: browsing issues briefly, I don't think this functionality is available just yet. Discussion of node selectors: https://github.com/kubernetes/kubernetes/issues/341#issuecom... -
For example we run ceph and use it in kubernetes. We obviously only run replica sets (the kubernetes feature, for those uninitiated) of 1 for these services. There's also some rough edges with locking for dead nodes, but it's definitely good enough for running the likes of Jira or Gitlab have single points of failure pretty much whatever way you scale them (unless you build a HA nfs server first).
Before that, we were running test clusters with just fleet, using custom bash and etcd for service discovery. Works fine, but you're probably better off with frameworks.
Now where things can go horribly wrong is you decide to use a clustered database, run it in kubernetes, thinking it's fine since you can't mount external storage for replica's but you have 3 nodes over 3 zones and you have backups. Kubernetes will quite happily schedule all of your replica's on the same host, meaning a single reboot and you're dead. There are hacks around this such as defining a hostPort, but of course if someone updates the RC you could happily be restoring to backup. You could run two multiple replica controllers to get around this and that'll allow you to mount storage again but that takes a way a lot of the elegance of the framework.
Point is, you can do it, but you'll need a bunch of uglyness and you'll need to be careful. If that doesn't sound great, way until the software properly supports it. :)