Distroless has debug images for this purpose: https://github.com/GoogleContainerTools/distroless/blob/mast...

Adding a shell seems antithetical to deploying production code as a static-linked binary, not to mention an expansion of the attack surface of the container.

Without a shell, how does one debug if anything goes wrong?

You can start a container with a shell that shares the PID and network namespaces of the container you want to debug.

Reading logs/traces on your log aggregation service and reproducing in a dev system?

How do you debug in the dev env without a shell?

from the host system, containers don't exist in a vacuum

With remote debugging?

remote debugging is a shell

not necessarily. e.g. java runtimes can expose debugging ports when needed that operate on a custom protocol.

or you can just build gdb into the container and run the process under gdb, then attach to the tty.

or you can debug from the host system where the container's pid namespace is a descendant of the root namespace and the other namespaces can be accessed via /proc or unshare.

What I meant is having a remote debugger is as good as having a remote shell in terms of remote code execution.

Debugging is about when the difference between theory and practice breaks down.

You can use nsenter