As sibling commenters are saying, the terrible experience that off-site workers have today is because companies haven't invested in the infrastructure to allow off-site workers to collaborate in the same way as workers in the office. Slack can replace the hallway conversations. Shared Google Docs can replace white-boards. Skype can replace meeting rooms. You just have to invest in the infrastructure, both physical and cultural to make it happen. And, as a side benefit, it's a lot easier to record the outcomes of meetings with these new tools, which makes it much easier to go back and reference what people said.
That would affect the remotes as well and doesn't set anyone up for a good working environment. If your desk were all wobbly, lighting dim and office space generally just lousy, you'd say the same thing about the office space and how it isn't conducive to a positive work environment.
I work remotely for an SF company and, honestly, I dislike making trips to head office sometimes. Why? Because the entire place is a distraction. I observe (and get dragged into) multiple coffee walks/chats a day that take up 20/30 minutes at a time. The constant distractions of an open plan office are something that office workers don't notice, but remotes do. I can get more work done in 8 hours at home that I feel I can do in three days in that office. I've spent most of my career in an office and only became a remote worker a few years ago, but it's clear to me now just how much of a distraction an office is. I would argue that this is already a more than justifiable benefit of having a larger remote workforce.
A large % of my colleagues are also remote, and it works great. Almost every meeting happens with at least someone on VC, and it works. In-person meetings happen in the office, but Slack has become our primary method of sharing ideas and having conversation. Ideas have definitely formed over lunch, but more of them form when there's an active conversation in a Slack channel.
All of your reservations are justified, though, because that seems to be the default thinking of many people who either haven't been remote themselves or who have had a less than stellar experience of remote working. I can honestly say that the majority of our remote force across different teams (eng, support, sales) are the best in the company. But remote working isn't for everyone, and there have been people who thought it was be perfect way to work, only to crash and burn because they couldn't handle it.
We need a cultural shift.
I just don't buy remote is the remedy to cost-cutting, it brings new drawbacks. Personally, I don't feel the same level of collaboration with my remote 'colleagues' than onsite ones. But it could be personal.