My dream solution to this issue is a one liner command like:

    docker exec -it --augment=ubuntu my_container bash

Would start bash in the container, but also layer into the filesystem all the rest of a standard ubuntu image only for my tools, but not affecting the application in the running container.

I'm pretty sure that's possible with current linux kernel mount namespace/overlayfs infrastructure used by docker - all that's needed is the command line tool to support it.

The new ephemeral container support in kubernetes lets you do essentially that. You bring the filesystem from another container image into the PID/network namespace of a running container in a pod.

lol, it's fun to imagine going back in time to explain what you just said to my 2004 sysadmin self. back when I used to build servers, and colo them, and physically maintain them.

Which version of bash would you expect to run in this example?

(1) If it's the bash version from the standard ubuntu image, you will need to specify where to mount your application's filesystem inside the ubuntu filesystem.

(2) If it's the bash version from your application, then it's the other way around: you will need to specify where to mount the ubuntu filesystem inside your container.

Option (1) seems more practical. My point is that you will need to specify a mountpoint either way, and your commands will need to take this mountpoint into account.

Mount both filesystems at the same place as an overlay - if a file exists in both, I don't care which I see.

I see. That will work as long as your app is built on the same distro as your tools image (in your case, ubuntu).

You can debug such containers by running another debugging container to join their corresponding namespaces. For example the most frequently used namespaces are pid and network, with these two namespaces joined the target container, you can see its pid and binary as well as the network traffic.

For docker and k8s, there are two helpful tools which implement what I said with simple and intuitive UI:

* https://github.com/zeromake/docker-debug

* https://github.com/aylei/kubectl-debug

Edit: Add links for the helper tools.

The github page for Docker-slim goes into some detail on how this can be done using a side-car container[0].

[0] https://github.com/docker-slim/docker-slim#debugging-minifie...

A good pattern is to build an image specifically containing troubleshooting tools which can be run and attached to a problem container's namespace. That gives you a standard set of tools without having to bake them into every image.

Is there more doc on this idea somewhere? In particular, being able to strace without root (etc) would make life a lot easier.

FYI, you don't need any special permissions to strace in a docker container - you just need to disable the default seccomp profile (docker run --security-opt seccomp=unconfined), which blocks use of many unusual-in-production syscalls including ptrace: https://docs.docker.com/engine/security/seccomp/

One common workaround floating around the internets is to use --cap-add SYS_PTRACE. This has the side effect of permitting the ptrace syscall, but it also gives you the ability to ptrace processes owned by other users etc. That's more than you need and it's kind of dangerous in a production-ish container.

I might be thinking of a different scenario (and I'm generally using Singularity rather than Docker). I want to start my container under 'strace' and see everything. This is not generally possible in the obvious way, as there's a setuid-root binary in the process tree that blocks further strace'ing.

(One can still attach after everything's running, but that's not always good enough.)

For running network related debug tool: use nsenter (https://github.com/jpetazzo/nsenter/blob/master/README.md) which allows you to run network related tools using the network namespace of the container.

For simple shell access, use the :debug variant of distroless images which include a shell.

For more complex troubleshooting, I think other people has recommended many ways. I haven't had the need to do such troubleshooting but if I need to I would mount an image with necessary binaries into the container. This is where distroless becomes handy: I can mount a Debian image and don't worry about ABI compatibility.

use a docker sidecar, in fact explained in the docker-slim repo:

https://github.com/docker-slim/docker-slim#debugging-minifie...

And with Kubernetes you can use Ephemeral Containers for that.

Answer to "How do you debug an issue in a running container?" is "you read logs or you don't."

Generally you build a special debug image that has busybox or whatever. In case of distroless, debug image has busybox and everything that comes with it.

Also, what are you trying to see with top/htop? In ideal world you will see a single process pid 1 that is your entry point. There shouldn't be more than one process running in it.

You can get resource consumption of a container without logging into the container just like you can get running processes without getting into container.

There is nothing else you can do without dragging wholelot of dependencies:

- Anything java related will require a JDK

- Debugging any native code will require a whole debugger

- Debugging python/ruby will either work or will require dev dependencies

Sidenote: Who the fuck uses ansible to debug containers?

yum install

I went to a presentation on Sysdig thinking that would be some kind of solution. Not really; not unless you want to hunt down or write syscall filters (or find some online) or pay for the Enterprise version.

I just wish there was a way to do the basics:

   1. Look at files within my running container (maybe even modify them, without needing vim or nano installed inside it).
   2. Ping/ICMP something from within the container (again, without ping being in the container itself)
   3. DNS lookups from within the container
   4. Connect to a port on an IP or DNS name from within the container
   5. Inspect the contents of a dead container that won't start without having to commit it first.

I did a post a while back on how I feel about debuggin within containers, and I should probably write another one because I don't think I cover those 5 things:

https://battlepenguin.com/tech/my-love-hate-relationship-wit...

For point one you can grab the running container tag and then add a layer on top with any tools you need.

You obviously won’t get the same operational state but if you want to poke around a container you’ve built and see what’s in it, you can just extend it.

I'd love to have 50 upvotes to give to this particular comment. With Docker being so fashionable, too many applications which have strictly no business being containerized are shoehorned into containers (e.g. Confluence).

My rule of thumb is: as soon as I have to `docker exec` into a running container because something's wrong, this container needs to be stopped and a VM should be used instead.