As for your comments regarding Solaris, I was using Solaris Zones about a decade ago and, inversely to your experience, I found the whole process really easy. In fact it was on Solaris where I first learned about OS containers and which really convinced me to adopt them to many other projects over the years since.
To be honest, it is really weird reading your comment as my experiences couldn't differ more. If your memory is up to the job, I'd be curious to read more detail about the issues you had.
So the way I used to use debian was I was using xen. If I needed something like a tftp server, I would bring up a tiny VM with nothing but the base debian system installed. Then I would simply apt-get install tftp-server-whatever (via puppet) and call it a day.
Since all it was running was sshd+syslog+postfix(bound to lo) and the tftp server, patching was easy. I didn't have to deal with managing updates for 2000 packages that I was not using. I could also upgrade each vm separately.
Zones did not work that way at all. You could do a 'sparse' zone, which would inherit some directories from the host system, or whatever the opposite of that was that would just go and install a full 4G+ solaris system into the zone. When I complained about the fact 'single use' zones had to have 4G of software installed, I was told to just use clones, which did not use extra disk space.
The problem is those full clones then needed to be patched.
It was basically impossible to create a small single purpose zone that did not contain 4G of software and was easy to manage.
The sparse zones were not usable because then you could not install or update software in isolation. I see the same thing on the ez-jail page:
> Because the basejail's copy of the userland is shared by the other jails, updating the basejail automatically updates all of the other jails. Either source or binary updates can be used.
So, if an update to basejail breaks something, it breaks every jail linked to it? What if I want to test my software on FreeBSD+1?
Really, the problem wasn't the zone technology, the problem was the horrible user-hostile solaris userland. This is why Triton appeals to people.
As would the VM's you said you ran previously. Sadly patching is one of the biggest problems in IT. Which is where tools like Puppet come into their own (which I think you did say you run?).
> "The sparse zones were not usable because then you could not install or update software in isolation. I see the same thing on the ez-jail page:
> > "Because the basejail's copy of the userland is shared by the other jails, updating the basejail automatically updates all of the other jails. Either source or binary updates can be used."
I think that's a little deceptive. You can still install packages in the containers separate from the basejail. ez-jails would be completely useless if you couldn't. basejail is essentially just the FreeBSD core (ie it excludes /usr/local).
> "So, if an update to basejail breaks something, it breaks every jail linked to it? What if I want to test my software on FreeBSD+1?"
Then you'd have that as a container without basejails inheritance. Nobody says you have to create every jail using the same templates (though you did earlier complain about having to patch everything in isolation :P)
Personally, I tend to just take a ZFS snapshot and then rollback if I ever run into issues. It's a little lazy but works well enough for planned maintenance windows.
-----
Those arguments aside you do raise some interesting points; and I hadn't heard of Triton either. So I've definitely learned something new today. I can't really say I've had the same requirements to spin up and destroy new containers with the same frequency that you have, which would explain the differences between our experiences. For what it's worth, ez-jail does have a solution to those kinds of usages via "flavours" (loosely like VMWare VM templates). However I suspect this might still fall a little short of your specific requirements.
> (though you did earlier complain about having to patch everything in isolation :P)
The problem wasn't so much that the zones needed to be patched, the problem was that they were full installs of solaris. Solaris didn't really have a concept of a 'base install', at least not without a ton of debugging and manual dependency management [0]. This meant that the only easy way to install a working solaris system was to all the packages [1]. This meant that you then needed to patch all the packages.
I think some of that got better in solaris 11(?), but I never made it that far. I think I switched to nexenta and then to debian/kfreebsd.
[0] I tried a minimal install, I got the kernel to panic by trying to enable nfs4+krb5p without having some separately packaged crypto kernel modules installed.
[1] Unless you were a solaris admin. The ones I asked said it was possible, but that I should just install everything.
It's not as obvious or well documented how to set it up this way, but it was maybe a couple days work, including my first rc script ever, and the script to assemble the jail in our environment (you could probably have a tar file with the contents of the jail, and just add the conf/data files)
If you know how to build a minimal chroot, you know how to build a minimal jail, and then you just need to configure it, etc... If you don't know how to build a minimal chroot, check out option 3 'most minimal chroot possible' here: http://ram.kossboss.com/debian-ubuntu-build-minimal-chroot/
It's written against Debian Linux, but the same thing basically works for FreeBSD, except ldd on FreeBSD omits the loader (/libexec/ld-elf.so.1), but I sort of remembered that from somewhere, and I used truss to see what system calls were being used before it died.
I tried to use zones many years ago..
For "isolated services" I mostly ran things like debian VMs with 128M of ram and 4G of disk space. In theory zones could do that too, but solaris was fundamentally incompatible with the idea of running a tiny 'container' with only the software that you needed installed.