SkyDNS looks interesting, but it doesn't appear to do any heath checks on the endpoint. I don't want clients to receive an answer to an A record query that contains the IP address of an endpoint that is down.
Am I mistaken?
You probably don't want a healthcheck done for every DNS request.
Better to have a healthcheck service doing healthchecking, and modify SkyDNS along with whatever else happens when a service goes down.
In order to be robust in the event of a network partition, the client should perform the the health check itself. This can be done in a background thread; it doesn't have to be synchronous with the DNS lookup (and that would be very bad for performance anyway).
Right now, if you use SkyDNS[1] as your DNS server, and attach Skydock to the Docker host, this all Just Works. Most people want to use etcd instead of Skydock, though, so support for that is coming soon[1] too.
[1] https://github.com/skynetservices/skydns
[2] https://groups.google.com/forum/#!topic/coreos-dev/iklEYHh5J...