> The thing about being able to have remote workers is that it works best when the company culture works in a way where everyone could be remote

Not really. It can make good sense for a company to generally insist on working from the office, but to make reasonable exceptions.

> How do you include them? By making everyone remote.

No, you just take reasonable steps.

> So now in order to accommodate someone you have to change the work environment for everyone

No, you don't.