Subtrace already works great on Kubernetes (https://docs.subtrace.dev/kubernetes)! Add a single line to your image's Dockerfile and that's it.

I'm working on an even simpler way where you can just `kubectl apply` a DaemonSet or a Helm chart to get automatic tracing for all pods in your cluster instantly without any code-level changes. If anyone is interested in beta testing this, email me at [email protected], I'd love to understand your usecase!

It’d be neat to use subtrace in an ephemeral pod for debugging purposes, that just runs alongside the regular pod.

For monitoring the network traffic for the whole cluster, the CNI and/or whatever ebpf-based runtime security stuff you’re using (falco, tetragon, tracee) is usually enough, but I can definitely see the usefulness of subtract for more specific debugging purposes. If run as a DaemonSet make sure to add some pod filtering such as namespace and label selectors (but I’m sure you’ve already thought about that).

> use subtrace in an ephemeral pod for debugging purposes

That's a great suggestion. It'd be like kubectl exec-ing into a shell inside the pod, but for network activity. I think I'm going to prototype this tonight :)

> pod filtering such as namespace and label selectors

Yep, Subtrace already tags each request with a bunch of metadata about the place where it originated so that you can filter on those in the dashboard :) Things like the hostname, pod, cluster, AWS/GCP location are automatically populated, but you can also set custom tags in the config [1].

[1] https://docs.subtrace.dev/tags

Disclaimer: I'm a maintainer of the project

Pixie (https://px.dev) can be installed in under 5 mins and gives this level of visibility across all applications. No need to change your application (wrap in `subtrace run`) to get instant visibility.

We also support 11 application protocols (https://docs.px.dev/reference/datatables/) with TLS handshake tracing and MQTT support coming soon (encrypted traffic tracing has been supported for a long time).

Might be an overkill to use it in this capacity but SUSE NeuVector (https://github.com/neuvector/neuvector) can build a connectivity graph between pods and allows to perform packet capture to PCAP. It works at the virtual switch layer and can capture all traffic.