This isn't a problem by itself, however remote companies need to have a lot more processes around information transfer and team building, which is something companies don't need to actively think about for the most part if they build on-site teams.
And what would you do if you are there in person? Challenge them to a duel?
This whole scenario sounds very paranoid to me. In my experience if you treat people decently and perform OK most co-workers won't talk negatively about you behind your back. Or maybe I just never noticed... it did have 0 impact on me though if it happened.
Someone senior who didn't quite understand some point but didn't want to ask about it in the meeting* will ask a version of it in the hallway just outside the door.
They'll appreciate and increase trust in the person who gave them the answer. There's not even anything nefarious about that; it's just normal human behavior.
* Positively: because they didn't want to derail the meeting and slow everyone down, or negatively: because they didn't want to admit to not being omniscient. Either way, it happens.
When I worked in a building, I would go to meetings, and there I would see co-workers and people would talk to each other in meetings. When I work remotely, I join virtual meetings, and I see co-workers and people talk to each other.
In both cases, people can use chat to say things without me seeing. If I am physically in a building, but people are not in the same room as me, they can talk to each other without me hearing.
The only premise that could be true is that within one organization, people that cannot be remote may resent people that can.