You can actually configure Docker to use an Unix socket instead of TCP[1]. It seems it still opens it up to every user (it runs a chmod 777 on it), but you can always enforce stricter controls (e.g. with SELinux).

[1] https://github.com/dotcloud/docker/pull/938