Debug containers have been a bit of a let down in UX. It's rather difficult to do sort of basic stuff, and a lot of stuff is hidden behind flags.

E.g., if you just sort of roll with the defaults, you're dropped into a pod in a very confused state: `ps -ef` says nothing in your debugee is running, and the filesystem of the debugee is nowhere to be found.

You can work around both of those (the first is --target, but the latter requires an intricate SO answer¹) but its the sort of thing that would be nicer out of the box?

The node debugging mode is a bit better: by default, puts you in at least the host pidns, and mounts the host FS.

¹https://stackoverflow.com/questions/73355970/how-to-get-acce...

As someone who doesn't do much (any) k8s...

Seems only marginally better than using nsenter on a privileged container to just go muck with the host

I've also been disappointed with debug containers. They are often not useful for debugging trick production-only issues because so many of those issues are related to container state, which can be (often is) different inside the container. Certain languages/platforms and developer discipline are better about this than others, like if you're using functional/immutable languages then it's less of an issue.

For applications that aren't super high security, I've been really appreciating using immutable hosts (that get regularly updated/rotated), along with CI/CD that is constantly rebuilding from source, applying latest software updates, and deploying the latest version of the app. Combined with other tools like scanners, and de-bloating your images, it really raises the height of the fruit.

> UX

I would agree, but so much of day-to-day Kubernetes is arcane CLI commands to begin with. Other stuff that is non-trivial to do on the CLI but comes up in most reasonable production deployments:

  * Rotating secrets without exposing the secret to the shell history file (hint: kubectl apply -f can take - to signify atdin, but not kubectl patch!)
  * Ensuring your edits to a ConfigMap pass application-level validation (i.e. your configuration changes won't crash your app, not just that it's a valid ConfigMap)
  * Anything to do with user auth or RBAC
  * Scaling the default persistent volume size of a StatefulSet

The truth is that Kubernetes is a platform, and just like how most people don't want to run a bare copy of Bash or VIM on their laptop, people will figure out aliases, one-liners, and other functions to help make them effective. So some of working effectively with Kubernetes means, yes, building your own custom debug containers, and writing your own helper shell stuff.

> without exposing the secret to the shell history file

Any command in shell with a space before it will be omitted from history.

Agree it should take input from stdin.

This requires `setopt HIST_IGNORE_SPACE` (zsh) or `HISTCONTROL=ignorespace`/`HISTCONTROL=ignoreboth` (bash/ksh) to be set and may not be enabled by default in many distros (e.g. NixOS doesn't, Alpine doesn’t, etc).

Always check your shell before assuming this will work!

I work on multi-tenant k8s clusters at CDNs and used to work at Rancher and have seen just about every multi-tenant / federated deployment there is, nasa, meta, etc. Stuff where even the hardware vfio paths mounting the nic or gpu channels keep users apart from one another, and the entire path out of the cluster are completely apart from k8s and any userspace and would be something like multus as a shim- what exactly are you referring to that can cgroup hop via a shell that we're not currently mitigating? It's the hardware being infected by something we worry about at this level.

A lot of the CDNs even use tools like kubevirt where the segmentation is even further. And then we have gvisor, firecracker, etc.

I admittedly haven't touched k8s code since 1.18 but I can't think of anything like you're referring to and I definitely would like to know about it.

Thanks.

Also can go even simpler and use apparmor and/or systemd hardening instead of containers

Please do a write-up of these debug features. I'd love to learn about them.

Have you read the docs? https://kubernetes.io/docs/tasks/debug/debug-application/deb...

Out of curiosity, how would you run Python or R workloads in kubernetes without a distro or shell?

Python is better than shell, so intruder will use it first.

Python needs an ld.so and libc (minimally) but not a shell or other external utilities. Shebang scripts are loaded by ld.so, not the shell.

Shebang scripts are supported directly by the kernel via the exec family of system calls, so ld.so shouldn't be involved.

https://github.com/torvalds/linux/blob/master/fs/Kconfig.bin...

  config BINFMT_SCRIPT
 tristate "Kernel support for scripts starting with #!"
 default y
 help
   Say Y here if you want to execute interpreted scripts starting with
   #! followed by the path to an interpreter.

Python with batteries included, doesn't that mean exploit tools included?

No personal attack intended, I am wondering this about my own embedded product which contains Python.

Yeah, I remember reading the code in the kernel that handles shebang a long time ago. ld.so is not involved.