Of course, you could have a local attacker get in through other means, and then access local DC traffic within your SDN. But if you get to that point, you probably have bigger problem than terminating SSL.
1) A disgruntled employee sets up a surreptitious tap on the network to see if any secret material comes through. A high value target would be `Authorization: Bearer` in HTTP headers, but there are plenty of other things to slurp up.
2) A normally honest employee running an unrelated network tap to diagnose an issue with the SDN spots your Authorization headers (or other secrets), and knowing that they have a legitimate reason to have the wire capture, copies out the key material. This is very hard to prevent, since network admins can and should be tapping the network from time to time.
I'm not particularly concerned about someone from outside Google breaking into the SDN fabric, though a hypervisor breach could leak network traffic from other tenants on your instance (if you are sharing).
You're sending traffic over google's SDN in the clear, which is still encrypted by google if you believe:
(The linked security design doc mentions 'encrypted in transit _to the data center_', but it doesn't address traffic inside the DC, last time I read through it in detail).
With a secret-per-pod, your key material lives in in the etcd on the API server, and gets mounted in a tmpfs on each pod that is given the secret. Only Pods in the Secret's namespace can access the Secret, so if you have RBAC configured correctly it should be possible to lock this down tightly to only the code that needs the Secret. (I'm not sure how to do this in GKE; I'm currently treating each cluster as a single security domain).
That might be OK if 1) your data isn's sensitive or 2) you're running on your own metal (and so you control the network), but in GKE your nodes are on Google's SDN, and so you're sending your traffic across their DCs in the clear.
There are a couple of pieces of hard-to-find config required to achieve TLS-to-the-pod with Ingress:
1) You need to enable ssl-passthrough on your nginx ingress; this is a simple annotation: https://github.com/kubernetes/contrib/issues/1854. This will use nginx's streaming mode to route requests with SNI without terminating the TLS connection.
2) Now you'll need a way of getting your certs into the pod; kube-lego attaches the certs to the Ingress pod, which is not what you want for TLS-to-the-pod. https://github.com/PalmStoneGames/kube-cert-manager/ lets you do this in an automated way, by creating k8s secrets containing the letsencrypt certs.
3) Your pods will need an SSL proxy to terminate the TLS connection. I use a modified version of https://github.com/GoogleCloudPlatform/nginx-ssl-proxy.
4) You'll want a way to dynamically create DNS entries; Mate is a good approach here. Note that once you enable automatic DNS names for your Services, then it becomes less important to share a single public IP using SNI. You can actually abandon the Ingress, and have Mate set up your generated DNS records to point to the Service's LoadBalancer IP.
(As an aside, if you stick with Nginx Ingress, you can connect it to the outside world using a Kubernetes loadbalancer, instead of having to use a Terraform LB; the (hard-to-find and fairly new) config flag for that is `publish-service` (https://github.com/kubernetes/ingress/blob/master/core/pkg/i...).