They do. They might be able to say that they don't care where their coworkers are (and I believe them), but when enough people are remote, their colleagues are forced into a remote working model too because the team needs to communicate in a different way now.

How people work is a mutual agreement, not something that's being done to one group by another.