Probably because things that don't get "used" can still affect things. There's a lot of detail in name lookup algorithms and SFINAE where computing what you end up using and giving each TU its own view is probably not much savings over using the headers as-is.