adlleong

If I understand correctly, one of the limitations of doing application level failure injection with Gremlin is that you need to integrate it into your code: https://www.gremlin.com/docs/application-layer/installation/

It might be interesting to combine these approaches and use a traffic split to send a percentage of traffic to Gremlin instead of integrating into the code directly.

That's right, and there are some interesting trade-offs. Having failure injection built-in is convenient but running a separate service gives you full control over the error responses. This can be useful if you want to simulate responses with error bodies, for example.

I'm the author of this blog post and I'm more than happy to answer any questions people have!

Running one linkerd per pod as a sidecar is definitely not crazy. You can see an example of this configuration here: https://linkerd.io/getting-started/k8s/

Using kubectl proxy takes care of locating and sending the service account token and certificates. This is described a bit farther down in the Kubernetes doc that you linked.