This isn't something I considered but it totally makes sense. Given that the Vary header is a per-resource value you would have to propagate that through the network. For millions of resources that might become an issue. And since in a worse-case scenario the server could be changing the Vary header for a single resource across multiple requests you have the additional problem of trying to keep it consistent across datacenters.

I think that is probably why some CDNs have a single configuration for any HTTP headers you want to vary on (e.g. Cloudfront allows you to specify a global configuration for a distribution that takes into account specific headers). This avoids the problem of both per-resource and inter-datacenter consistency that relying on the Vary header might cause.