The problem areas — and the reason you’ve basically all got to be in the office at some point — are decision making ones where a group of two or more people fervently believe in two or more mutually exclusive ways forward. Even when I worked remote for a company multiple timezones away, I would be in the office one week every quarter to first build social capital and later to cash it in when building consensus.
I also think back to one of my stakeholders was an underperforming SWE intern. The rate of online engagement that best suited them was zero, and yet — as with any intern — it was crucial to us that they left feeling positive about their time here. That stuff is really hard over the phone.
And you force everyone to commute into the office.
I've been working remotely for 6y; we have multiple external business partners, and a team of over 50. We have team members and business partners in Russia, Finland, Sweden, Germany, UK, France, NYC, BC, Cali, Japan, and elsewhere; we operate without core hours. I, personally, regularly interact with people in all those listed locations every day.
Never once has there ever been a need for an in-person meeting. Never.