>I long for the day where Kubernetes services, virtual machines, dedicated servers and developer machines can all securely talk to eachother in some kind of service mesh, where security and firewalls can be implemented with "tags".

That's exactly what Scalable Group Tags (SGTs) are -

https://tools.ietf.org/id/draft-smith-kandula-sxp-07.html

Cisco implements this as a part of TrustSec