Yes, you do downsize salaries. Paying Jill less than Bob where Jill is a better worker because she lives in Little Rock Arkansas and Bob lives in NYC is downsizing Jill's salary.
Jill can do better. Hopefully other remote companies seek out the Jills and pay them what they are worth.
We don't downsize your salary. But sometimes we can't upsize it. There's a few companies we can't compete with on salary like Google. Most of those employers seem to be in the bay area. Sometimes we lose out. A lot of times we don't. The world isn't quite so black and white here.