They key is the lack of friction. If all the employees are in the office, your one remote outsourced person is the outlier and teams will get annoyed having to deal with that person. If everyone is already remote, then there's no pushback at all from having another remote person -- and if they're already remote, why not hire from somewhere cheaper?
Communication patterns matter.
In the office, much communication is implicit or at least not committed to electronic format: "hey, can we grab an hour to whiteboard some stuff?"
In a fully remote environment, everything should be captured in an async friendly format. As other comments have mentioned, it's much easier to scale that communication mechanism to employees from different areas of the globe.
Commuting daily to some horrific open-office and sitting there with your headphones on isn't going to save you when the beancounters see that they can get an engineer at the same level in [cheap country] for 1/3 of what you cost. Even if having an employee sitting 30ft away is worth something, as you said, is it worth being able to 'hire 2-3 engineers' for the same price?