> As a platform for running production code it might be dead or dying, but as an ecosystem and a development tool it will continue to live and probably thrive.

Not everyone needs an over the top Kubernetes cluster in production. I'm plenty happy using Docker Compose in production and foresee myself continuing to use it as long as Docker maintains it. There's even a WIP issue on their roadmap[0] to rewrite Docker Compose with Go to make it more consistent with their main CLI. Looks like full steam ahead to me rather than death.

It's really nice having the same Dockerfile + docker-compose.yml get used in dev + ci and prod. There's no surprises and no massive amount of complexity.

[0]: https://github.com/docker/roadmap/issues/15

I second this, we've been running Docker Compose in production for years and love it. It's massively simplified our production environment, and for a company that only serves 100,000-1,000,000 monthly hits it's the perfect middle ground. We've tried k8s several times and have always ended up going back because we didn't need the complexity.

We even run compose v2.4 to retain the ability to set container memory/cpu limits and define startup order with healthchecks because we don't need the complexity of swarm and v2.4 provides everything we need. Sometimes simple is best.

I personally use compose to manage services on my own laptop and homelab as well, it just has the massive advantage of being the greatest common denominator that everyone can pick up in a few days if they're familiar with basic container concepts.

>>> We even run compose v2.4 to retain the ability to set container memory/cpu limits and define startup order with healthchecks...

Does compose finally support starting containers in order with healthchecks? Last time I checked it didn't and Docker Inc was expressively refusing to support that, in spite of being a very basic use case.

See various workarounds over the years: https://stackoverflow.com/questions/31746182/docker-compose-...

Compose v2.4 was released long ago and abandoned by docker. Some features/workarounds that worked in that version were killed in the next versions. It seems to me that you're stuck on that old version unable to upgrade because of that?

IMO this is a perfect illustration of how Docker sucked in practice (both the product and the company), the industry naturally converged into trying to get rid of it because there was no other choice.

You just define a depends_on: condition: service_healthy, works great in v2.4. v3 is really a different beast, it's not an upgrade in the traditional sense, more of a lateral move with the introduction of swarm. I've even seen Docker employees recommend in forums using v2 independently of v3 if you need certain features (I can try and dig up a link from my archive if you're interested).

Compose file version 2 is not deprecated, unlike version 1. It remains stable and supported (though no new features are added), it's very well documented and still widely used because of the additional features it supports over v3. https://docs.docker.com/compose/compose-file/compose-version... (compare with the v1 deprecation notice above it)

I don't know about "sucked in practice", we use it because we love it. They released a new version to push users toward swarm, which we don't need. Such is life sometimes. It's had some bugs and problems in the past and if something better comes along we'll use it, but for the time being Docker compose satisfies all our needs and I think it's a beautifully designed UX and reliable piece of software in its current form.

Problem being, "depends_on" was removed in version 3, with no replacement in sight.

I honestly don't understand why they'd remove that? why make a new configuration format just to remove critical options? what is the developer supposed to do instead? when will compose stop accepting the v2 format so we're definitely screwed for good? (yes the writing is on the wall).

P.S. For readers who don't have context. Make a basic compose file with a webserver and a database. "docker-compose up" is failing half the time out-of-the-box because the web server starts before the database.

Docker compose development was compromised by the agenda of pushing people to use swarm.

This is why v2 features like resource limits were moved under the deploy: key and you have to use the special "--compatibility" flag to get them to work locally.

This is no longer the case. Compose is an open spec and the v2 and v3 formats are being merged into a unified format (where you don't need version numbers). There are now multiple implementations (eg for ECS/Fargate and ACI), and implementations are being encouraged to support all the options.

As of the September release of Compose version 1.27.0 https://github.com/docker/compose/releases/tag/1.27.0 the config format versions 2 and 3 are merged and "depends_on" – and "extends"! – are back.

This is amazing news, I was not aware of this. Thank you so much for linking! Do you know if it also applies to the cpu/memory limits and health checks? Is it a true full merge of the two specs? (I will investigate myself and post back too)

> "docker-compose up" is failing half the time out-of-the-box because the web server starts before the database.

Most web frameworks will retry until it can connect or give up after a specified timeout time. Basically handle it at the app layer where you have the most control.

But you still run into situations where you need to wait for docker-compose up -d to be "really" up before doing certain things (such as exec'ing into your main container to run tests or migrations).

For that I wrote a simple Bash script at https://github.com/nickjj/wait-until. IMO a problem like this can't really be solved at the Docker Compose level. But without such a script you can easily end up with failing CI tests because there's a multi-second but varied amount of delay on upping a database for the first time. That script has become a part of all of my CI / deployment pipelines.

Wait for a TCP port to come up or wait for a HTTP healthcheck to succeed. I think it's exactly the sort of things that should be supported by the tool out of the box. It's a very common need.

The job of the tool is to manage a bunch of containers that depend on one another and restart them on failure. The tool has to know about dependencies and liveness anyway.

This is an application-level design flaw, not a flaw inherent to docker-compose. Regardless of whether you use docker or even if you use containers at all, your webapp should be able to restart if the connection to the database fails or is interrupted for any reason.

You would have the same problem if you used supervisord for example. The general solution is to auto-retry the connection until it comes back up, or hard fail and auto-restart the entire app in the event of a hard failure (e.g. with restart: on-failure or autorestart=true).

Hm, yes, but to play devil's advocate - if you want compose for orchestration (I don't) then you want that dependence not so much for initial start up, but for triggering restarts on failure.

As it stands, AIUI, all it does is mean that dependency A comes up too (in some order) when you `up B`.

You can define any order or restart behavior you need with v2. As long any container with a service dependency fail hards if that dependency is missing/down, it will be restarted and the whole app will eventually reach an up state.

    restart: on-failure
    depends_on:
      other_service:
        condition: service_healthy

https://docs.docker.com/compose/compose-file/compose-file-v2...

https://docs.docker.com/compose/compose-file/compose-file-v2...

From GP:

> Problem being, "depends_on" was removed in version 3, with no replacement in sight.

How is v2 relevant?

v2 and v3 specs are merged as of September apparently, so either set of options works in the current version. (See cousin comments above)

Exactly this here as well. I have helped setup a few successful SMEs with Docker, Compose, and some CI/CD best practices. They are running 7-figure businesses on the backend.

Couldn't agree more. compose is a wonderful tool that lets you avoid the k8s complexity if you don't need it.

I use Docker Swarm (basically multi-node Compose) in production, and am also really happy with it. It's an order of magnitude simpler than k8s, and perfect for small-medium scale deployments.

In terms of configuring your stack, you use regular Compose files, with a few enhancements available (such as better support for secrets). I love it!

I also want use docker swarm in production, but I keep hearing people say it has network bugs like after some time services cannot talk to each other. Have you experienced any such issue?

I haven't experienced any networking issues in production at all.

I have experienced 1 networking issue, but only specifically when keepalives are disabled on Windows (where I use Docker for dev/test only)[0]. When using an overlay network, network connections to dockerised Postgres go "stale" after 15 minutes. I workaround it by publish the Postgres port in "host" mode instead of the default of "ingress" mode using `endpoint_mode: dnsrr`.

[0] https://success.mirantis.com/article/ipvs-connection-timeout...

What's your solution to the problem of shared disk between hosts ?

I've never actually had to use shared disk, but I'd use NFS if I did (as another commenter suggested) - I've tried it out before, just when playing, and it worked fine.

Nfs volume mounts. Works quite well. Make sure the hosts can access the nfs endpoints (i run nfs on separate lan)

I recommend taking a look at Hashicorp's Nomad. I was in a similar situation where I wanted container orchestration but our use case wasn't yet at the point where the complexity of kubernetes was justified. Nomad strikes a great balance. Its single binary and makes it easy to get a cluster up and running. You can run tasks other than docker containers as well.

Compose plus VS Code’s remote container plugin gives a brilliant development env, and easily deployed to production. I agree with your assessment!

There’s no need for them to live on. There are open source alternatives that mimic Docker exactly. In fact on Fedora[1] the “docker” cli command is actually buildah and podman (you actually can’t install Docker on Fedora anymore - I genuinely haven’t noticed a difference). The commands are exactly the same right down to the command starting with the word “docker”.

I don’t wish them ill, but literally everything has been replicated. If they fail as a company it won’t matter to developers who don’t use their enterprise software.

I actually think the company isn’t long for this world. It wouldn’t surprise me if they got bought for a paltry sum in the next few years.

Their lasting legacy is that most software and people assume that a shortened container image points to docker.io.

1 https://fedoramagazine.org/docker-and-fedora-32/

Red Hat removed docker prematurely, Podman is not ready to replace docker for all production cases yet.

We have had to replace RHEL systems with Ubuntu to keep some of our applications running as they fail on Podman. A major problem is also that podman-compose is far from feature parity with docker-compose.

> We have had to replace RHEL systems with Ubuntu to keep some of our applications running as they fail on Podman.

I replaced Centos8 with Amazon Linux 2 ("amazon linux extras" provides a working docker installation) for exactly the same reason. I like the idea of podman and buildah and I wish them well, but for the time being I can't afford to debug the missing pieces and subtle failures.

True but I would use pod yaml with podman. It’s not anywhere near being on par with Docker-compose but it’s the better tool because every pod can be used rootless.

I'm aware of podman, which is more or less a drop-in replacement for Docker, but is there anything comparable to Docker Compose and Docker Swarm?

I use Docker Swarm (technically "Swarm Mode") in production, and I'm really happy with it - it's so easy to configure and operate, but I am worried that it will no longer be maintained at some point.

You're not wrong to worry (scroll down to "What about Docker Swarm?")

https://www.mirantis.com/blog/mirantis-acquires-docker-enter...

I use Swarm at home and for my projects because the cognitive and resource overhead is so low. It just works and it's easy to set up.

buildah/podman/skopeo are nice tech. and all, but they don't exactly mimic the setup of Docker for Windows/Mac.

If for no other reason that, they don't do Windows containers :) I know of quite a few corps using Windows containers in prod. now, take up may have been slow, but it's happening.

Also Docker for Windows/Mac have an easier install/setup process than manually setting up VMs and installing tools on them.

> There’s no need for them to live on.

Maybe not, but considering how much they gave, I think we should want them to live on, and do what we can to help them. Do we really want to live in a world where a startup company makes as big an impact on the way we develop and deploy software as Docker did, only to have all possible business taken from it by bigger players?

I understand this sentiment, but a lot of the original team and the certainly the original founder aren’t even there anymore. Not to mention that a lot of Docker’s business missteps have been self inflicted wounds.

There have been lots of great technologies that haven’t directly made their creators rich, but their good legacy usually/eventually catches up with them.

> but their good legacy usually/eventually catches up with them.

I think that's the just-world fallacy. [1] It's up to us to reward good work. So I guess I should go get a paid Docker Hub account.

[1]: https://en.wikipedia.org/wiki/Just-world_hypothesis

So I agree with you about supporting good technology which you want to see survive.

I don't think the poster you're replying to was invoking a just world fallacy, however. I read it as saying that developers who build solid systems that people like tend to thrive, even if their business fails.

That appears to be true, and not because of karma, because top-tier developers are in very high demand. Demonstrating that you're one of those isn't the same thing as running a successful business.

Podman is 1:1 compatible for running hello world. Your mileage in real projects will vary.

podman in podman doesn't seem to work. that's a showstopper for a lot of work.

I had had some pretty positive experience with docker-in-docker running in podman for ci/cd until podman v2 was released. Then I started getting a lot of weird crashes and segfaults and ditched podman.

Yes, you read it correctly, running privileged docker-in-docker containers with podman to spawn docker containers in dind containers. The good thing is that networking was handled flawlessly. And there were no conflicts between docker and podman

podman claims to be a drop-in replacement for Docker, yet when I tried to treat it as one, I ran into bugs and a lack of feature parity.

Had the same experience and the compose equivalent was barely functional.

Yep, same. We have had to replace RHEL with Ubuntu for container workloads to keep things working.

Docker Desktop is the only non-hacky way to get a decent Docker (and k8s) setup for development purposes on macOS and Windows. Linux on the desktop never needed that, but those other two do.

Have you tried WSL2? It's been a dream for my use cases as a web developer.

> you actually can’t install Docker on Fedora anymore

Moby is docker effectively. Your link describes how to install docker on Fedora.

Podman is not mature yet and it has many bugs. I am currently using it for a project, and lately I am starting to doubt it is prod ready. But it has some security options not available with docker. So Docker is still my first option.

Are you absolutely certain? I had to hack around a bit but I got it running just fine: https://ibb.co/X7dpt7g

A bigger issue is the fact that they were not able to become a unicorn despite having a tremendously popular product. It means there is no money in development tools innovation. This is not a good thing for the software community as it means all tooling innovation will only done and controlled by the FAANGs.

I work for Red Hat but I'm a long time fan and user of Docker.

I don't think that's what this means at all honestly. Docker (the company) made some poor decisions that led to their troubles. I'm not here to crap on docker, but they quit investing in and moving their tool forward because they were focused on the EE offering. This opened the door for "competitors" to do the things docker wasn't doing (daemonless, rootless, cgroups v2, just a few). They also did things that scared people, like the whole "moby" rename thing. It made people worried that (fairly or not) Docker the company didn't have the best intentions for the open source version. They had poor direction on product. They would routinely refuse nice features that people wanted (many that included PRs) for things with terse, unkind "do not want" rejections, and then 6 months later they would end up adding it anyway. Some of these were suspected to be refused so as not to allow open source docker to compete with EE feature sets. docker-compose was the worst at this (To be fair I think it was mostly one very loud/very powerful person wrt docker-compose).

Docker started the race way ahead of everyone. To this day people still use the name "docker" to generically refer to containers.

Anyway, my point is not to crap on docker, just to say that in my opinion, in so many areas important to an open source leader, they made decisions that undermined their long term outlook.

for me it seemed:

- they stopped evolving the Dockerfile. Really this is the complexity killer that developers just "get".

(also they wouldn't have their entire API cloned if they were moving it forward)

- they subtly force everything through docker hub and conveniently forgot to do something like let people have a local repo. (redhat lets you define a local repo with --add-registry)

- telemetry. I went to install it on macos and it started collecting telemetry the moment I launched the installer.

Great points. I think they started requiring an email and sign up too for the Mac download, although if you had a deep link that would still work but if you didn't already have it you had to sign up.

There's no money in backends unless you have a great enterprise sales team.

SAS, Matlab, Oracle, Arc, there's lots of really boring backend systems with plenty of FOSS alternatives but you need sales.

If the customer is the developer, then they might ask their manager for a text editor with a nice UI. If the customer is the whole enterprise, then someone needs to wine and dine the CEO to get that $10,000 per core contract signed.

This is a really interesting point. I work in backend enterprise software (as a consumer more than a provider).

Surely the critical success component at scale is more than just "wining and dining". I truly wonder how Oracle has managed to do it for all these years...

There's also developer education (see mongo db), support, tooling, ecosystems ...

Because the FOSS generation feels entitled to be paid while refusing to pay for tools, which leaves the typical enterprise customers as the only place one can make money with development tooling.

That's not true. As a FOSS lover, I pay for a lot of tools. Both development related and unrelated. Moreover, some of the tools that I pay extend on FOSS software (Tower, Viscosity, Permute) and some of are FOSS already (Cyberduck).

The thing is, as a person I don't want to feel ripped off. I don't subscribe to JetBrains since my work doesn't require these tools (also, I use Eclipse for 15+ years and it works well). The subscription culture allows developer to pump features indefinitely but, I can't subscribe to all tools that I like.

There are many services that I subscribe too (Dropbox, Spotify, Netflix, IFTTT premium, Evernote, etc.) but, they give me a service which touches my life everyday. I have no budget to subscribe to a tool to test for a year. It's not feasible. I need to eat.

So yes, I'd rather have a good FOSS tool which doesn't want $100+ from me every year and I'd rather patch it myself. But, if you provide me a good service which I can't replicate easily, or don't want to manage myself, I'd pay you good money. I'd be also very happy if the tool I'm paying is FOSS.

Also the biggest obstacle driving me away from paying is the tools are closed source and I'd be locked in if the tool/company lets out the magic smoke. All the services I pay are not locking me in. I can get my data out of them. So it's not always wanting to earn money for free.

In my opinion, the Jetbrains tools are very affordable when compared with the huge productivity boost that they grant me.

I agree that intellij and eclipse are similar, but as soon as you work with C, C# or Python, there's very few good open source alternatives.

I have no doubt about their quality and affordability. Also, the price goes down when you subscribe for a longer time.

I write C/C++ and Python mainly. Eclipse CDT and PyDev are very good. Since I'm using the platform for 15+ years, I've seen its bad days too. Except some edge cases, CDT is bulletproof and works very well. PyDev is also very smart and helps me the way I need.

Actually, Eclipse has the best and most sensible Git UI I've ever used. I also like Tower but, Eclipse both makes sense and helps a lot in the relevant places.

Eclipse CDT is very good? I think we have a very different definition of very good. Could write a long rant on everything that was broken in Eclipse for C and C++ development.

One question on eclipse. Did eclipse finally sort out git/svn support?

Here's an interesting tidbit of history for those who don't know. One of the main drivers for the decline of eclipse was the complete lack of support for source control, which is a pretty basic feature for an IDE.

You could try to get some plugins. There were 2 major plugins for SVN and neither worked half the time for no particular reasons. It was a complete shitshow. Don't even think of doing that in a company where HTTP access must go through a proxy.

Developers eventually gave up and moved to JetBrains IDE. Source control worked out of the box.

Eclipse CDT was really bad back in the day. Slow, crashing, not smart enough but, it's very good now from my point of view. It can satisfy my needs and, I'm happy with it.

Heck, I've developed my whole Ph.D. with it and it interfaced the tools that I need to use well, was stable and fast. When I pressed a shortcut, it did the thing I expected it to do. The whole experience was, well, uneventful.

I've used Subversive IIRC during my Master's and it was uneventful too. Didn't lose any data, the plugin didn't misbehave or had any problem with it. Was using Assembla's SVN + Redmine bundle as my remote repository.

The git support was similar. I just installed it and it worked. Still works. As I said before, they've nailed the best logical view for git IMHO. IIRC, now it comes bundled with the Eclipse.

They may have done some things wrong in the past but, it's a solid IDE and I like working with it. I don't think they deserve the strong words you choose, but to each his own.

Of course, your mileage, taste and views may vary.

I wonder if a timed open source license would work? One in which code/features were automatically scheduled by the license to revert to GPL/BSD/whatever at a point a year or three in the future?

I think that might sap some of the impetus to reinvent those features in a competing open source project (they can of course just copy them if they wait), and also might entice people to pay that wouldn't normally because they don't want to get on the paid product train. Users also get a known date a feature will be available, a real date, not a projected delivery date.

In a lot of ways it's what many companies that develop products do already to allow people to get familiar with their product (have an open source version with less features), but this allows them a better story, makes users feel more sure about what's going on, and if they release the code immediately but it's unusable for a period by other projects, that does make it harder for those projects to cleanly reimplement unless they're sure they haven't seen in (that might be a net negative for the public with a litigation happy company).

The hard part would be tracking the time, but git has ways to make that pretty simple.

A number of databases uses the Business Source License (and its variants, usually including a non-compete clause). It acts as an option which vests (switches to a non-copyleft libre license) after n months. There is also the Polyform project but adoption has been low and their licenses are extremely controversial and unpopular.

I don't think that's a good read on the situation. I think a better analysis is that there is no money being what amounts to a facade on top of an operating system/kernel capability. All the real novelty and challenge to containers is in the OS itself. Everything else is more or less ease of use, which is important but relatively easy to replicate. In the span of 5 years or so, Docker went from being critical glue code to cruft.

That facade enabled millions to have easy access to that particular kernel feature. BSD jail's have been around for decades without mass adoption.

BSD jails weren't widely adopted because the BSDs themselves weren't widely adopted by the time the industry got to the point where something like Docker was increasingly seen as necessary. I remember being involved in one project that had to decide this exact question in the very early days of Docker - jails were much more stable back then, but customers wanted Linux containers, not "sufficiently Linux-like".

I think the GP's point is that alternatives can easily spring up when the main feature-set/value is in the kernel rather than the glue code.

It seems that HashiCorp does their business well, what makes it?

Take this post with a hint of bias as it's written by a Red Hat'er - if you ask an employee what the "next thing" is, I wouldn't be surprised to hear "podman" (which is an opinionated docker clone)

The number of Docker Desktop installations and the popularity of local Kubernetes tools (think KinD, k3d) show that "Docker got boring" which is an aspiration for infrastructure tooling.

Author here. I no longer work for Red Hat these last few years, and my words are strictly my own. I wanted to communicate what's happening now in the context of what I experienced from Docker's early days in the industry.

Podman would be next big thing if: 1. they make a package like docker ob mac where you dont have to fiddle with manually starting vms. An environment where using docker feels native 2. When they start the vm to run podman they will use qemu on arm to simulate an x86 machine and run x86 containers

The second one will be a huge step forward as docker will not be able to do this for a long time as they use the mac hypervisor which can‘t run x86 vms. Podman would enable again prod/dev comparability which may work for redhat to more people switching to podman even for their production systems.

Agreed, podman needs better support for mac. It works fine under Windows WSL2 currently, see: https://www.redhat.com/sysadmin/podman-windows-wsl2

The Podman team is working on it but it's not quite there yet. Podman now has an API so at a minimum the whole opaque VM thing should be usable here soon. podman-machine and boot2podman already work for some people, although they didn't work for me.

I have a dream of using a lightweight Fedora Core OS VM on Mac/Windows for running the containers with a podman-remote client seamlessly driving it from the Mac user space. Once podman-remote is working, that's my next plan ;-)

A self plug here, I'll be keeping an eye open for all of these things and will blog about once I get it reliably working. I'll be submitting it to the Red Hat developer blog but I cross publish everything I write to medium as well in case you want to look for it: https://freedomben.medium.com/ I'm also starting to tweet when I blog (I'm finally getting serious about it now haha) @Freedom_Ben on twitter.

A lot of people seem to think 'running under WSL2' is equivalent to 'works on Windows', but they're just not the same thing. That's like saying 'it runs in a VM on my Mac'.

Thanks for sharing because this is interesting to me. I would not have guessed that. When I use WSL2 it's mostly transparent to me so I don't think about the implementation.

Do you consider "Docker" to be "works on Windows?"

> Red Hat'er

Hat'ers gonna Hat.

(disclaimer: author does not actually work for Red Hat.)

Back in 2013 before I took a job at amazon but after working with a game dev for playstation I had 2 months off, playing with docker I realized how good it's be for crap like chroot jail dev or multi-project dev env isolation. There were big blockers for it being used this way then. I did some chatting with the docker team and we did push things a little in that direction. But the vision was still about running in prod.

To their credit the docker folks I talked with did listen and respond and move the needle (obviously) on it being a local dev tool, so I cheer them for that.

I use it all the time in my windows box for dev. Tho I do try and keep my software able to run on any os.

Their collaboration work on integration with WSL 1 and 2 has been fantastic.

What's your workflow with WSL2 and docker for Windows?

Not that poster, but I use Fedora in WSL2 and the Docker for Windows integration and things (and by things I mostly mean `docker-compose`) just...work.

They have some k8s stuff I've never looked at, too.

I wish we could replace Dockerfiles already with a format that allows for fine grained layer control.

What kind of control are you looking for?

I'd like explicit layers (transactions, I guess?) - so instead of every command making a layer, I could write

    FROM alpine
    STARTLAYER
    RUN apk update
    RUN apk upgrade
    RUN apk add foo
    ENDLAYER

and end up with a 2-layer image (alpine + my stuff) rather than the 4-layer image that docker would produce today.

I have always wanted that too.

the way to create small docker image is:

  RUN apt-get install foo && \
    do bar && \
    do more && \
    remove all of the crap you don't want

It should be replaced with actual first-order dockerfile statements

You can use multi-stage builds, but they are sort of hacky and do not match your thinking.

You can also use --squash but it is a hack too and usually is not available where needed because of experimental.

Also, ADD is stupid.

You can add a tar.gz file and it will expand it (but you can't use -k)

You can add the same file using http/https, except it WON'T expand it so back to the RUN dilemma you get:

  RUN apt-get install wget && \
    wget foo.tar.gz && \
    tar xvkf foo.tar.gz -C dir && \
    rm foo.tar.gz

I too have wanted a solution for the massive amount of layers that are generated just by doing normal things. When you reach a point where you're trying to be so clever just to avoid the creation of a useless layer, it's the tooling's fault, not yours.

You might investigate buildah. You can achieve precise control over layers. Also, buildah mount is awesome; direct access to a container file system from the host; COPY/ADD et al. are obviated.

Have your tried multi-stage builds? We use them almost everywhere to slim down images to only have what they need in production.

https://docs.docker.com/develop/develop-images/multistage-bu...

Yeah we asked for transactions and they gave us this... thing.

It’s cute, and useful, but in the end it brings back the primary problem that Docker actually solves: absolutely getting all of the files your code needs to run into the package, and the right version.

Multistage builds require you to airlift files out of one image into another. Accurately. If cherry-picking is the best option on offer by docker, there are plenty of other tools that can do that.

Collapsing layers is about seeing that an image provides a set of services at point A, and another set at point B, and that nothing in between represents an interim state of any substantial value. For most images, this is one or two layers that relate to either the main payload, or one particularly volatile dependency.

The real purpose for multi-stage builds is for compiled languages.

If you have some service written in say C++/Java/C#/Go, then building it in one container (that has the build tools) and copying the build result to another that only has the stuff actually needed at runtime makes good sense. That is much easier and saner than trying to uninstall all the build time components after using them.

It is not like for desktop software we install the build tool, build, and then uninstall build tools. Instead we grab the relevant build outputs and package them into an archive or installer.

But as a solution for clearing out cruft like package manager caches, etc, yeah multi-stage build is not really optimal.

Being able to write the executed steps in a natural way, and then specify at the end to compare everything to come specified previous layer, and create a layer that just has the differences would be really useful.

They could even still cache the intermediate layers as part of the build caching process if they wanted, so long as they are not included in final image as pushed to a repository.

Docker does have a super cut down version of this with its experimental `--squash` option, but that only makes sure there is a single new layer for the output of the whole dockerfile run. You cannot use this if you for some reason want exactly 2 new layers. This limitation is probably acceptable in a lot of cases. However this option is not supported in the buildkit based backend yet.

RUN --mount=source=my_script.sh,target=/tmp/script.sh /tmp/script.sh

This will adds the script without adding a layer for it.

The appropriate solution for this is to put your shell code in a shell script and RUN that.

Then you have to create a layer for sticking the shell script into the container and another to clean up after yourself.

If the Dockerfile is canonical, I should have as much as I can reasonably put into it.

Isn't this the same?

RUN apk update && apk upgrade && apk add foo

In the simple case, yes it is, but it scales poorly; consider ex. https://github.com/docker-library/postgres/blob/b9c080857b88... which is 112 lines run together like that to keep things in a single layer. This is pretty bad for readability. Also, as sibling comment notes, that only works for RUN commands, not other or mixed groups of commands.

It is, but the syntax is uglier. Also imagine that you might want to do more than just RUN in a single layer. Maybe COPY some file into the container and then RUN something that uses it.

Spend some time on docker hub and you'll find thousands of Dockerfiles with RUN <150 lines of && concatentated scripts>. It's dysfunctional.

Good to know, I often wondered why people produced these giant RUN statements instead of making multiple RUN.

It keeps living on Linux too. My customers are too small to need Kubernetes and we won't switch off our docker containers in development and production. And a customer created a container on his Mac on Thursday to run something I run natively on my Ubuntu laptop.

To be fair to the author of the post, he wrote about the continuing life of docker in the inner loop of development. It will keep going also in production on medium and small projects until we start using something else for multiplatform pull and run software distribution.

Take a look into NixPkgs / NixOS

Phew, thanks for that. That article got me all agitated, thinking a core piece of tech I use is on its death bed with no alternatives.

Not necessarily, I am yet to bother using it, and know plenty of environments where Docker still is a foreign word.

Containers should be about security not cloning local development environment.

> Containers should be about security not cloning local development environment.

Why not? One way to use containers is as a light-weight virtual machine without a kernel, which can be pretty useful.

A workaround for broken development practices.

Sandboxes are about security. Containers are about deployment tooling. Some containers happen to include/use sandboxing techniques.

I have used container based technology for about 20 years now, starting with Virtual Vault on HP-UX, and happen to know a couple of things about what containers are supposed to be.