Thanks for mentioning these.

I've been using Alpine religiously for years, until the build problems became too big. Mostly long build times and removed packages on major version updates.

Now I first try with Alpine and if there is the slightest hint of a problem, I move over to debian-slim. So things like Nginx are still in Alpine for me, while anything related to Python not any longer.

At first I thought your mention of docker-slim was an error for debian-slim, but I've followed the link am glad to have learned something useful.

From the docker-slim README:

    Node.js application images:
        from ubuntu:14.04 - 432MB => 14MB (minified by 30.85X)
        from debian:jessie - 406MB => 25.1MB (minified by 16.21X)
        from node:alpine - 66.7MB => 34.7MB (minified by 1.92X)
        from node:distroless - 72.7MB => 39.7MB (minified by 1.83X)

Why are the minified Alpine images bigger than Ubuntu/Debian? Are a bunch of binaries using static linking and inflating the image? Or something else?

You can try using the xray command that will give you clues (docker-slim isn't just minification :-)). The diff capability in the Slim web portal is even more useful (built on top of xray).

I have quit using Alpine. It caused to many issues in production. For some workloads in GO for instance you can use Scratch directly. But slim and distroless are my preferred base images.

Personally I hate scratch images because once you lose busybox, you lose the ability to exec into containers. It's a great escape hatch for troubleshooting.

There are a few options that help here. With host access, I tend to just use nsenter most of the time to do different troubleshooting. It can be a bit of a pain doing network troubleshooting though since the resolv.conf will be different without the fs namespace.

And kubernetes has debug containers and the like now.

Yes we have experienced many issues with Alpine too. Ubuntu with 25MB compressed is ok.

Until it's not. Well working deb/apt is replaced by some self-sabotage called snap before up-to-date packages land there.

Snap is horrible.

The software required for running a Snap Store instance is proprietary [0], and there are no free software implementations as far as I know. Also, the default client code hardcodes [1] Canonical snap store, so you have to patch and maintain your own version of snapd if you want to self-host.

Snapd also hardcodes auto-updates that are also impossible to turn off without patching and maintaining your own version of snapd / blocking the outgoing connections to Canonical servers, so snapd is also horrible for server environments. To top that, the developers have this "I know what's good for you, you don't" attitude [2] that so much reminds me of You Know Who.

[0] https://www.techrepublic.com/article/why-canonical-views-the...

[1] https://www.happyassassin.net/posts/2016/06/16/on-snappy-and...

[2] https://forum.snapcraft.io/t/disabling-automatic-refresh-for...

Yep. I am trying my best to boycott Canonical and their closed source Snap which is akin to Apple Store or Play Store, but for desktop... for... LINUX. Goes against the philosophy in every way imaginable.

It's really sad especially given how Canonical introduced so many people to Linux through Ubuntu. I understand they need to monetize to survive but I wish it wasn't like this. I miss the "Ubuntu One" service, a simple Dropbox like alternative that you pay for. Completely optional and server side. Integrated into the UI.

That being said, I was wondering how many people actually find the Snap system and ecosystem useful. Reverse engineering snapd (which is licensed under GPLv3) and snap app format in order to create a compatible server would be a fun project.

So,another systemd in the making... how long until snap takes over dns resolution...

Say what you want about systemd, but it is still Free Software.

Are people actually using Snap in containers? It feels like a convenience feature for desktop users.

They don't give you the choice. Applications are only avaliable as either a snap or an deb. With many you might want to containerise (ie Chromium for CI) being snaps. I dont belive they can work in unpriviledged docker containers.

There are Chromium debs for Ubuntu 20.04 available from the Linux Mint repo.

To get it Chromium under docker, I'm using the following params:

  --headless --no-sandbox --disable-gpu --window-size=1920,1080 --disable-dev-shm-usage

Which apps are you using via Docker, that are only available as a snap?

I understand that, it's just my impression with using Snap that it's used to ship desktop applications in a consistent way.

you cannot use snap inside a docker or any other OCI container, first of all snap is a containerised package as well so it doesnt make much sense but what is more important it requires SystemD and as far as i know if systemd isnt PID 1 snap deamon wont run and its CLI will output it cant run.

> I have quit using Alpine.

As I said to the author of TFA a few days ago: Alpine delenda est.

What issues did you experience with Alpine and Go?

What go build options are recommended to have go output a static binary (that can run inside a scratch image)?

If you're using plain Go, you get static binaries for free. If you're linking against so C library you might be out of luck. You can try setting CGO_ENABLED=0 in your environment, but I've had mixed success in practice.

I’ve only had success with CGO_ENABLED. If you depend on a C library, then obviously it won’t work, but mercifully few ago programs depend on C libraries (not having straight C ABI compatibility is a real blessing since virtually none of the ecosystem depends on C and its jerry rigged build tooling).

Well this is not what I'm seeing. I need to add -ldflags "-linkmode external -extldflags -static" to the go build command otherwise the binary doesn't run inside a scratch image.

CGO_ENABLED=0 has always worked for me in production, but it did come with some tradeoffs, notably, network performance.

Layers support in docker-slim is something that's on the todo list (exact time is tbd).

The recent set of the engine enhancements reduce the need for the explicit include flags. Also new application-specific include capabilities are wip (already added a number of new flags for node.js, next.js and nuxt.js applications).

Could anyone ELI5 how exactly docker-slim achieves minifications of up to 30× and more? I've read the README but it still seems like black magic to me.

basicly what docker-slim does it basicly checks what your program is opening/ using(using simular system as strace) and what it does open/use is then copied to a new image. And how can get those kinds of numbers, basicly it removes parts of rootFS that are not required which is basicly your base images standard files like /etc /home /usr /root..., it also removes all development dependancies, source code and other cruft you might have copied in for use during build or simular.

While absolutely genius, it would be more awesome if we could shift this to the left. And have dpkg or apt, or something new, only fetch and place those binaries that are needed.

while in theory possible it would require a whole lot of change.

Also like kyle said apt and alike(all current package managers) cant generaly deal on file per file, basis package is as is all files included, so this would require a new package manager(or old one) with packages being a single file with no dependancies for specific package. This would require new base images if we used dockerfiles and you would need to know every library every binary your program needs. While on other hand leaving in all the cruft some copy in like READMEs LICENSE files and so on.

Benefit of docker-slim is that it is in many ways just a line in your already predefined CI/CD pipeline, therefore its just another step likely just working with already preexisting technologies you might use in your pipeline.

it'll be possible to do something like that in the future where docker-slim will generate a spec that describes what your system and app package managers need to install. Using the standard package managers will be tricky for partial package installs though because it's pretty common that the app doesn't need the whole package. Even now docker-slim gives you a list of files your app needs, but the info is too low level to be usable by the standard package management tools.

So nasically program footprint as a namespace? Hrrm.

While this is the go-to approach, I find really hard later to debug problems for images who don't have even an shell or a minimum set of tools to help debug a problem

I recommend avoiding this kind of thinking, which leads to bundling all sorts of stuff in every single container. My philosophy is that images should contain as little as possible.

To debug a container, a better way is to enter the container's kernel namespaces using a tool such as nsenter [1]. Then you can use all your favourite tools, but still access the container as if you're inside it. Of course, this means accessing it on the same host that it's running.

If you're on Kubernetes, debug containers [2] are currently in beta, and should be much nicer to work with, as you can do just "kubectl debug" to start working with an existing pod.

[1] https://man7.org/linux/man-pages/man1/nsenter.1.html

[2] https://kubernetes.io/docs/tasks/debug-application-cluster/d...

Related, is there any built-in facility that would log file system accesses that failed on the resulting image? Seems like that would give the answer in a substantial fraction of cases.

you can use --include-shell to inclue a shell. but my recommendation is always keep a non-slimmed image somewhere so you can refer to it and some debugging can be done in either, also that allows you to use something like slim.ai's(company behind docker-slim) web based SaaS features which allow you to see the differences between 2 images and see the file system in nice web base file tree. some bugs may stem from removing what is necessary for app to work(sometimes caused by not loading a library during "slimming" process) for those types of errors you need to know how your app runs more then anything. it a tradeoff

The problem is that sometimes, using commands like cp don't output the error in case of failed permissions and if there's any tool into it (e.g. ls), you can't find out why it fails

Typically I just use Go in a scratch base image where possible, since it’s super easy to compile a static binary. Drop in some certs and /etc/passwd from another image and your 2.5MB image is good to go.