Istio delivers encrypted packets across services/ pods but in the pod, envoy proxy does the decryption and puts the packet unencrypted back on the interface to be consumed by the application.

In kubernetes, for calico CNI, the pod networking is done using veth pair. One interface remains on the host to be hooked to the host bridge and the other interface is sent to the pod namespace.

Mizu is deployed as a daemon set(one pod per node) and with enough privilege you can "tcpdump" the pod interface(the one where the envoy proxy puts back the packets unencrypted) (similar to "ip netns exec")

Basically, they tap the pod interface from host level. (They have a blog post about this as well: https://up9.com/gain-visibility-into-istio-mtls-traffic-with...)

This is my understanding so I might be wrong

How much latency is this going to add?

It sounds like it's not adding anything on the path, that's just another process sniffing the interface. If that process doesn't read fast enough, I assume it will miss packets but not slow things down.

Welcome to Kubernetes!

I don't know what went wrong but the ecosystem prefers software presented as a black box and shrouded with mystery.

If you contrast this with Envoy, it has stellar documentation -- Envoy wants you to know how it works!

Envoy is used and abused by Istio (k8s wrapper) which has terrible documentation, history of horrific upgrade/update story, (& software quality).

That hasn't really been my experience, I have found an abundance of documentation (sometimes getting lost down the rabbit hole). But I have kept clear of the corporate buzzword solutions, that might be why.