The services in question are built with a hodge-podge of shell scripts and build tools, so getting them all to compile locally is a challenge, let alone deploying them. My hope was that containerizing the builds would isolate any configuration problems, and that containerizing the deployed services would cut down on outages by permitting trivial rollbacks (say, by snapshotting all the service containers before each deploy and merely restoring them should a deployment fail). Of course, all of the above could be fixed by traditional means (e.g. rewriting the build system with a single, standard tool; streamlining the deployment process, etc.), but it seemed like Docker could solve 80% of the problems while easing the implementation of the proper solutions down the line.
Considering the above, do you still think Docker's a poor fit for business that aren't building hosting systems? Oh, and any nuggets of wisdom you could throw to a newcomer to the industry? :)
You don't have a standard repeatable way to set up an environment now. You need to do that first before jumping on docker I think. Once you have that, you can start replacing parts of the setup with docker and see if it fits your needs.
The advantage of ansible is that it is idempotent and the changes it makes to the system are the same ones you make manually or via bash scripts. So it is quite easy to debug
If you come to a new place and "there's an error in here somewhere", the difference between layers of images held together with shell scripts and a Ansible/Puppet/Chef script is like night and day.
Something like the following
docker run -v `pwd`:/tmp/buildresult your-weird-hodpodge build-command
Among other things, I see docker as an extremely useful mechanism to decouple server maintenance (build server and deployment server) from the tool maintenance. It can dramatically help reproducibility, etc. docker run -v `pwd`:/...
you're no longer running pure Docker, but Docker + a shell script. Those shell scripts bloat horribly, are not portable, and are a pain to maintain. This is precisely my problem.As for rollbacks, the exceptionally bad Docker tagging system just adds headaches to rolling back efficiently. If your production OS has a package management system, consider building packages for that - after all, it will have been battle-tested and known to work on that OS. There will be a learning curve for any packaging system, but using a native one means less faffing around later - remember also that docker is changing a lot with each release.
Also, as mentioned in the article, logging with docker is difficult and hasn't been solved properly yet, and if you like production logs for troubleshooting, Docker requires some attention before you can get those logs. My devs just run the app and watch STDOUT... which isn't easy to log in docker. Then, of course, they complain that they don't have production logs to debug, and subsequently complain when I ask them to modify their logging so I can slurp it :)
Anyway, Docker is not a packaging system for use in-house; if you're only using it to package stuff... you will be ripping it out later on down the line (this is what happened to me). On the other hand, if you open-source your stuff and want to provide 'canned images' for random members of the public to use, then there is a point to using docker, since you don't control what those host machines will be running.
In short, Docker is a complex ecosystem with it's own learning curve, and doesn't really save you from learning curves for other things. If you can't articulate the exact problems that using Docker will solve for you in production, I would advise against it.
Edit: If you need a standardised provisioning system, start out with Ansible. It's pretty straight-forward. Admittedly I've only used it and Puppet... and puppet is better aimed at large/complex infrastructure environments.
I quit the job.
The scenario played out just as you said: I ended up single-handedly and poorly re-engineering something that already existed (they did have a working Ansible setup) for no visible gain. "Swimming in containers upon containers" is exactly what happened; they kinda worked, but the farther we got, the more kludges piled on top of each other. In four months work we didn't even hit production - the most we got was a CI/QA service that was actually nothing more than a loose bunch of Python scripts. Between managing dev/test/prod differences, tracing missing logs, removing unused volumes, networking all that stuff together and trying to provide at least a decent level of security, I realized that I'm wasting everyone's time and money. Developers hated it because it filled their workflows with traps and obstacles. Admins hated it because of the lack of tooling. Business hated it because it caused unexplainable delays. The only thing we really accomplished was some compliance with the The Twelve-Factor App - something that could've been done in a week. Hardly a victory.
My advice? Forget about Docker unless your primary business is building hosting systems. It will take years before Docker gets mature enough for production, and not without a ton of tooling on top of it and some major architectural changes. Until then, go back to the old UNIX ways of doing things... it worked perfectly since the Epoch and it will continue to work long after the 32-bit time_t rolls over. You'll be fine.