I can't stress this point enough. If you don't know what's up with a failing service, any SRE can go into the Dockerfile definition (actually, we're rarely dealing with Dockerfiles anymore, these days it's more the Kubernetes definition), look at which ports are exposed, what the environment variables are, what the service is talking to, etc.
You can harden security between those services much tighter than between binaries, by dropping capabilities, seccomp, network policies, read-only file-systems.
Also, all the other advice about the drawbacks is pretty dated: While I've run into my share of super annoying Docker problems myself in the early days (and yes, especially on Ubuntu, goodbye Device Mapper...), I've yet to encounter a single, Docker related bug on Google managed GKE on Container-Optimized OS since we've switched all of our services there a year ago.
I didn't regret our choice in a very long time.
Ding ding! Applications are only a part of the issue. It's the whole server stack, all internal communication, configuration, service compsition, and networking setup.
Having these things in a declarative format are a godsend for production, a godsend for mixed language environments, and a godsend for whoever gets to kill those services and replace them with better things.
Working with sensitive data: there is also the whole shit-show of secrets management... It's not like this is impossible without containerization, but having formal security boundaries around them (both in process and systems), gives you guarantees instead of well-intentioned guesses.
Containers are ran on VMs. Most of those VMs are ran in a cloud operated by the same providers peddling the container is new cloud. AWS/GCP/Azure building block is not a container. It is a VM. VMs are not hard. VMs can be treated like containers. Use them. Tool around them. You would be amazed how less crufty your stack would be when you don't use abstraction on a top of abstraction.
Containers gave us a rise of super flaky software architecture. The one that theoretically should be horizontally salable but is not.
Whenever people talk about how containers did the same to the apps as to shipping i tell them they are absolutely right. There's a pile of hard science(tm) used to properly load containers on the ships based on the content of individual containers and the trip they are going to take. That's hard science is sorely missing from the app level container management
Re: VMs vs Containers -- the implicit assumption in what you're describing, and in the article, is that you're building in AWS/GCP/Azure with engineers who are competent in that specific suppliers ecosystem and relevant server techs. Working with hybrid cloud installations and "air-gapped" systems with international partners invalidates a lot of those assumptions... Having a stable cross-section for deployment and scaling across clouds, backed with holistic answers to configuration-heavy parts of the application stack, is a total game changer. Containers don't let you ignore VMs any more then VMs let you ignore hardware, but trying to recreate Kubernetes ad-hoc in-situ is a crap-fest.
As a software engineer I gotta disagree about abstractions creating cruft. That's what poor abstractions do. Good abstractions moves complexity to it's most reasonable point of management and the lowest point of pain. "All problems in computer science can be solved by another level of indirection" ;)
I have spent two years watching people moving broken architectural design to containers because the consensus was "containers would solve our problem". When the containers did nothing of sort the next level of solution was "K8s would solve the problems of the containers that we have". That also did not work. We are now waiting for a better version of service discovery/up/down wiring to come to K8S.
Container is an abstraction of shared hosting. Application that works well in containers is the same application that works well in shared hosting environment, on a bare metal or on a VM. Do that. Make your application work well in a shared hosting environment. If it does, then it will work great in any kind of a container, be that a chroot, a docker container or a VM.
I absolutely agree that containers let get one started with a real work sooner, but so can rails app template generators or express application builders in nodejs. If this is the class of problem that people advocating containers try to address then I'm absolutely, positively for containers everywhere.
> Having a stable cross-section for deployment and scaling across clouds, backed with holistic answers to configuration-heavy parts of the application stack, is a total game changer. Containers don't let you ignore VMs any more then VMs let you ignore hardware, but trying to recreate Kubernetes ad-hoc in-situ is a crap-fest.
Theoretical ability to run an application across clouds because it hooks up into Kubernetes is a pie in a sky promise. If your application even marginally complex, there's no bloody way it runs on AWS and Google without an ocean of the blood (load balancers work differently, VPNs work differently, firewalls work differently), sweat and tears of the SREs and old grumpy developers who don't care about container specialness, but can look at the logs and write some additional special proxy-wiring-of-this-to-that-on-a-container-using-this-kind-of-adapter code.
> Containers don't let you ignore VMs any more then VMs let you ignore hardware, but trying to recreate Kubernetes ad-hoc in-situ is a crap-fest.
That's not what containerize all the things people advocate. Their claim is that container is a magic bullet that allow incompetent developer produce a production code. And technical management is buying it because you know... Docker.
Except for Tryton (SmartOS).
He also trivializes the massive cost and difficulty of switching stacks. Even if a company can afford to upend their development processes to use a different language for new development, most of IT is legacy technology, and dynamic languages have been popular for long enough that legacy Python, Ruby and PHP applications are part of IT estates, just like legacy C++ and Java applications.
I cant take this article seriously, its just some person refusing to adapt because "in my days we did it differently"
I think you paint the whole world with the bleeding edge brush. Many fortune 500 companies are a decade behind the technology stacks described in the article and in this discussion.
Hybrid cloud installations to leverage cloud computing resources are becoming more and more "the norm". In that context having apps and services pre-packaged for deployment in heterogenous environments is strategic power.
"The cloud" tends not to be cheap, but using cloud services to carry out jobs your dedicated site never could hope touch is something with business value.
And in regards to scripts or docker. No you wouldn’t need to rewrite everything.
Web development is full of engineering astronauts who think they need 8 levels of abstraction to solve relatively simple problems. Oh, and for that 0.1% that does actually switch, they soon realize they're abstractions weren't as good as they thought they were and they have to reimplement every interface anyway.
In regards to setting up a new server for a new provider, or docker, you have some boiler plate:
-> create script to attach docker to provider
New provider...
-> create new script to attach docker to provider
----
-> create script to provision server on provider
-> run generic script to configure it how I want
New provider
-> create new script to provision server on provider
-> run generic script to configure it how I want
In this regard, moving from 1 provider to another, is identical.
How do I abstract this unique idea of IOPS that AWS has?
So how do Hetzner, OVH, Rackspace, and hundreds of other companies stay in business?
Hint: cloud services are in fact way more expensive on every single metric. But they are more flexible.
And if you know all your servers have the same Python version and your code is pure Python, you can even forget that and do a .pex (https://github.com/pantsbuild/pex), which a the Python equivalent of Java .war : a zip that embeds all the program and dependencies. It's light, fast, and problem free basically.
No way. Just package your libraries with your executable:
/my_path/ld-linux-x86-64.so.2 --library-path /my_path /my_path/my_binaryI’ve done what you mention with Docker and alpine builds (for alpine Python wheels, for example) but honestly it’s a lot more painful to ssh into running containers when figuring out the build script than it is to just throw up a clean VM.
I understood the point that it may be an overkill for static binary deployments, but how is that dangerous?
Docker is an unifying interface. It's great at what it does: make a standard facade for app deployment, setup and service management.
But it also comes at a cost. And as usual in the IT industry, the people using the tech sell it like a silver bullet, and avoid to tell you that:
- docker is complex as a product and a concept
- knowing how to run docker != knowing how to use docker properly. Remember the nosql debacle where people though they could not formalize their schema because the DB didn't enforce it ? Geeks love the circle of life.
- docker is rarely alone, it will come up with an ecosystem that will creep in your stack. Like you never really do reactjs alone but soon find out you have 1000 plugins for a complex build system and you architecture depends on a flux implementation
- most things just don't need docker. Dockers solves a technical and organizational scaling problem. If you don't have the problem, you don't need docker. Just like you probably don't need a blockchain for your new client reward policy.
- docker is a LOT of new things to learn. Not only how to use it, but also how to do all the sysadmin things you used to do right, again, but with docker. Network management used to be a fun one. Not running as root again. Persisting data.
- you will need something to install docker and the infra anyway. So you will not drop Ansible/Chef/Whatever. They will work together.
- people use docker as an excuse to not do their job properly. No packaging, little doc, etc. "Just use the docker image". This one leads to bad times.
- Not everyone uses docker. You don't live in a bubble.
- Docker images are big huge binary you download from internet and run on your server. It's very painful to audit. And so easy to ignore.
- The paradox of choice. So many things in the docker world. And it changes so fast. It's really hard to know what to use, and to keep up.
I've seen many projects wasting time and resources just because of the docker setup. It's like those people setting up a distributed architecture for a job that takes 10 seconds to run as a bash script.
Bottom line, if you think you need docker, don't use it. People needing it don't think they need it. They know they need it. And it's useful for them because they use it right.
- No packaging, little doc, etc. "Just use the docker image".
I'll take a project with a dockerfile used for CI over a project with short setup instructions. The first one is less likely to go out of date, or be incorrect. (Ideally it would have both, and a good operations manual. But given only those two, very common choices)
- Docker images are big huge binary you download from internet and run on your server.
I agree this is how most people use it, but you're not forced to use it that way. You can deploy memcached based on 200MB Ubuntu base, or a custom images containing the lsb layout + libc + few devices, totalling ~8MB. The huge images are a documentation/community issue, not a technical one.
Well, we did it that way for 2 decades without Docker, and the world hasn't collapsed, so?
All of this is possible without tools like Docker, but they're a hell of a lot harder without some of the facilities brought by tools like Docker. It doesn't have to be docker, but it does need to be something offering isolation, a mechanism to manage images, and ways of managing data volumes. You can do that in bash if you want to, but the value in these tools is not to have to reinvent it from scratch.
I can't help thinking that Docker is building a massive legacy application headache further down the line when all these applications will not be developed anymore, the developers will have moved on, the technology stack moved on too to new shiny tools, but the underlying OS needs to be updated and these legacy apps still need to run. The compatible base image will probably not even be available anymore. Some poor guy is going to have a horrible time trying to support that mess.
That is largely what Docker is providing. It doesn't do it perfectly, but it does do it.
> I am can't help thinking that Docker is building a massive legacy application headache further down the line
That "legacy application headache" existed before Docker too, and it was much worse, because you had to hunt down toolchains and underlying OS images that'd let you get the apps to compile or run, and you could often not rely on the build and installation instructions to be complete and precise enough, because you had no guarantee that nobody would manually interfere with the process. This is in my opinion the biggest thing Docker's focus on automated image builds have given us: if you have a remotely sane process, the Dockerfile serve as evidence that the build steps work unattended (yes, you can do this with CI to verify your build steps too... until they fail in production and someone applies a workaround and don't roll it back into the build scripts; it's possible to add all kinds of steps to avoid it, or you can just build to images - docker or otherwise - and prevent the problem entirely by overwriting all he static data on every deployment)
> The compatible base image will probably not even be available anymore.
If you have a running copy of the image, then Dockers layering mean you have the compatible base image. Won't necessarily help you that much when you want to upgrade to a newer version, but at least it allows you to inspect what the dependencies are.
A docker with no coupling between the host and guest OS would make me happier. Then you have a binary you can just deploy 10 years after it has been compiled. And then you could extend this model to all kind of apps, including client/UI apps.
> But all you have done is to move the problem further away from the ops guys.
No, what you've done is ensured that you have the full chain available, so that you can do your upgrades, test them, and deploy them first when they're fully tested, instead of having to try to upgrade production machines without knowing if it'll work. It doesn't save you from untangling dependencies and version incompatibilities on upgrades, but it ensures that you can do that offline, and that when you're ready to upgrade, you have images you know works.
> Even if you have the source code the tools to compile it probably don't even compile on today's OS versions (if you can even get the right licenses).
You don't need to, as long as you retain the images of the versions the tools do run on. This is why I maintain most of my build environments in images too, so that I can retain not just the finished images, but the images used to build them.
> A docker with no coupling between the host and guest OS would make me happier. Then you have a binary you can just deploy 10 years after it has been compiled. And then you could extend this model to all kind of apps, including client/UI apps.
Sounds like you want a VM. Though Docker container come close in that you're depending on very narrow interfaces. I've got Docker containers running that were made from OpenVz containers that were built from machine images back in 2008. They're still running, if only because the original machine images were poorly documented enough that rebuilding them would have been a pain. But packaging them first for OpenVz and then for Docker worked just fine with minimal effort. I'm not sure what more you want.
so you mean ... reinvent kubernetes?
Not the clusterfuck that is kubernetes.
What you would want is an interface that doesn't require that coupling.
POSIX is probably something closer to what I mean.
As the Joyent branded zone is the native instantiation of the OS sharing the same kernel, there is no hardware virtualization, or rather, hardware is virtualized only once and zones then share the abstraction, as they are just normal UNIX processes. It's a completely different approach than hardware virtualization. One global zone can have up to 8192 non-global zones all sharing the same kernel, and the imgadm / vmadm combination makes the Docker problem become a non-issue.
Note that Solaris Zones predate Linux cgroups by several years, and that Zones offer full process isolation and were engineered explicitly for isolation/"OS virtualization" purposes, whereas cgroups were initially an attempt to give the kernel information about how to share resources between processes, and still do not offer real security separation or process isolation. cgroups is also a moving target, as "cgroups v2" or "unified cgroups" is still just barely becoming usable (CPU controller just added in 4.15, which is only a few months old).
You can run Docker images directly on SmartOS, because Joyent implemented the Docker API, and they also implemented a Linux compatibility API. It's not always perfect, but the fact that it works at all is admirable.
SmartOS also contains the equipment for hardware virtualization, and Joyent has even ported KVM (and recently bhyve) to illumos.
You can, theoretically, manage OS and hardware-virtualized systems at once through SmartOS, similar to what libvirt tries to provide on Linux (though their LXC integration is deprecated iirc).
I am not a SmartOS fanboy and have not as yet deployed it into anything more than a lab setting, but it, along with anything else non-Linux-centric, is consistently ignored for no real reason.
Well, if it ain't broken, don't fix it.
Is "sort of worked" better than "only works until the next version of docker and docker's next new, new storage layer"?
If you need to add 100x the moving parts and hidden interactions to solve some hard problems then your hard problem just got harder. You've just kicked the can down to production.
To add: this is wrong from another perspective. There are more goals than "massive scale" in software development. Your main costs are wages, so anything that allows programmers to be much more productive (or cheap!) is good for your business. Therefore, settling on something like Python makes sense if you can leverage its ecosystem and the wide availability of the relevant skillsets. I won't be switching to Go until its ecosystem and popularity can compete with older languages.
For instance, it's not particularly hard to wrap everything from the app down to the ruby interpreter in a single directory that you can then drop into a .deb, but because deployment stories in the Ruby community, from at least Capistrano onwards, focused on getting the app code onto the server and then working from there, with all the problems that entails, the folk knowledge of how to do it just never spread that far. You had commonplace craziness like using RVM to install Ruby on the production servers because it was convenient, and once that mindset was embedded, the community as a whole (and I'm tarring with a very broad brush here, but it's not inaccurate) ignored simpler solutions that worked better, and didn't need tooling with as many points of failure. Bundler is another case in point: you just don't need it in production. Or at least, I never found it necessary past the build pipeline, and I think a lot of folk never found out whether they did or not.
The point is, a lot of the apparent deployment difficulty in interpreted languages is entirely incidental and self-inflicted. Now, there is a real argument that Docker removes the need to learn all the ecosystem-specific folk knowledge you need to get to a point where you can do painless, sensible deployments in a new environment, but I think it's just moving the problem around.
But that's not the only way to do it; it's in keeping with the "uberbinary" idea to have a single tarred up directory and a known entry-point, such that once that's copied to the server you don't need anything else, no post-install build steps to get running. I built something to do everything-but-the-interpreter a while ago: https://github.com/regularfry/au.git.
The tricks that let these things work almost always boil down to knowing about the existence of a very few environment variables, and when in the build process you need them set to which values. `au` relies on $GEM_HOME, $GEM_PATH, and $PATH, and knowing what to do with $DESTDIR and `ruby-install` ought to get you a ruby interpreter as well (although `ruby-install` is just a shortcut to `make` in the MRI source). Where you'd look to learn that stuff now, I don't know.
Read this again. It make total sense.
Eventually, some path become running against the current, and then the better path is move to something else.
And what about that millons-of-lines of code? Is the same thing as "what about all the investment in money with that inneficient car that eat gasoline like is not a tomorrow, I push so much on it but the new tech is far more efficient?, and you tell me, let that in the past?"
Yes.
You put that millons-of-lines in the past and rebuild. If you have the correct mindset the change will pay off GREATLY...
P.D: I do this for work several times in my life, With very drastic/radical changes.
In fact, the MORE RADICAL the BEST. "Small" changes are not worth it. The $$$ is not there. But when you do a hard change is when the benefits get more evident...
Just from an ops perspective, if I can use Ansible to provision infrastructure, configure bare metal, configure VMs and build containers (Docker/LXC/LXD)...all of a sudden life gets easier - especially for keeping the containers themselves up to date.
Containers get rid of all these problems: Your images are immutable, you can always build from scratch, and you have full control of what goes in the image. Because of this, a plain shell script or something like Dockerfile is much simpler and easier to reason about. I really don't see the point in Ansible Container.
Do you typically deploy all of these technologies into one VM in production?
I don't think I understand the scenario being suggested here. (I'm experienced with Ansible but not Docker or Kubernetes)
However, I think the author trivializes the amount of work required to make different types of Python/PHP/NodeJS/whatever apps all work in a consistent way through configuration management, saying "I can just write a bash script or makefile." or "just download roles from Ansible Galaxy". This is so painfully ignorant and irresponsible that I fail to take the article seriously as a whole.
Even if it's just a Jenkins job doing a docker pull && docker run I've still seen massive improvements in the maintainability of configuration management code because there are so many fewer edge cases you have to take care of. No K8s required for that.
The author, writing about languages like PHP and Python (and I assume NodeJS becuase it also works in a similar way, but he doesn't explicitly mention it for some reason):
> They arose from an old paradigm, which is not suited to a world of microservices and distributed computing in the cloud.
So just because they can't build a single static binary, they're not suitable for microservices and distributed computing. Got it...
> If the switch away from Ruby/Python/Perl to a newer language and eco-system allows you to achieve massive scale with less technologies and less moving parts, then you absolutely have a professional obligation to do so.
Wow it's that easy?! Be right back, re-writing all our code in Go over a weekend hackathon.
This article made me angry.