This works on any language. I only use scratch in prod. Even for nodejs or python... compile a static interpreter binary and truck on.

Dev tools like bash, ls, grep, etc, have no place in production and only increase attack surface.

How do non running utilities increase attack surface? If you're able to execute inside the container couldn't you just write whatever utilities you want in?

There are many exploits that may give one the ability to execute shell commands. If there is no shell or commands to even write a file in the first place, mobility becomes limited.

Most container workloads I have use a read only root fs.

once in awhile i use 'kubectl exec ' to run some commands against my running prod containers to debug something or extract an environment variable. can i still do that without a shell or anything?

k8s has first-class support for ephemeral debug containers that share process namespaces for this purpose. Pretty cool feature imo.

https://kubernetes.io/docs/tasks/debug/debug-application/deb...

That sounds sweet! But I guess DigitalOcean doesn't support it? I'm getting "error: ephemeral containers are disabled for this cluster (error from server: "the server could not find the requested resource")."

You can’t, no. One approach is a “debug sidecar” container in the Pod that has your desired tools. Linkerd, for example, will add such a container if an annotation is present on the Pod: https://linkerd.io/2.11/tasks/using-the-debug-container/

I usually just include a shell, but to each their own.

You'll need to make sure that the binary is statically linked though.