I've worked remote for a half dozen years now, and you're absolutely right, the lag is definitely the major factor. Minor factors are just the extra little effort expended in making sure your screen is showing, making sure everyone can hear you, the screen jumping focus as it follows the speaker. Just things like that add to it.