In the case of Cassandra I'd have certain cassandra.yml tweaks I need to make, possibly different for beta/staging/prod, and docker-cassandra needs that config file. I also need to spawn the containers with the right setup, such that the right ports get exposed and shared volumes get mounted so the configs get read in. The spawning is a slightly different ops problem (do I want a homogenous cluster where I just spawn containers somewhere as needed, or do I want certain containers on certain machine types), but it seems like something that would fall in the domain of a provisioning tool.
Should I just be building the configs into the container and rebuild the container if they need to change? The isolation of the process seems at odds with my desire to have a semi-dynamic, centrally managed configuration.
I'm also considering that Chef itself is overkill to manage Docker containers and there are other tools more well suited to managing Docker-based infrastructure, but I'm not sure what they are.
https://github.com/kelseyhightower/confd
Basically, you add confd into your Docker image and execute it at runtime to do just-in-time config file generation. Here's an example for nginx:
https://github.com/grosskur/nginx-confd-dockerfile
You can create separate environment variable files for beta/staging/prod and pass --env-file to "docker run".
This lets you use the same Docker image across all your environments and avoid the operational complexity of mounting config files in Docker volumes.
We've been using chef in the past to manage docker, which never felt very good because the main cookbook was somewhat buggy (but it has gotten better recently).
We now use mesos+marathon(+chronos for deployment) for running our docker container, which sounds much more overkill than chef, but turned out to work much better and cleaner.
I'm using (b) for our nginx configs, you mount only the single config file as read-only, then let chef copy up the config before mounting/launching.
Docker pretty much removes the the need for such complexity. Don't think of it as a provisioning tool, like chef. Don't think of it as a VM. Think of it as an isolated filesystem and process environment that eliminates the complexity of supporting multiple, disparate application stacks.
The storage for persistent storage with Docker is less compelling, because now you have shared resources between containers that should be isolated. That said, we run Cassandra clusters in Docker and have no trouble.
Also, If you want to re-use the work you've done with Chef, there are chef plugins for Docker which make it easy to provision Docker images with chef (solo, I think)