If you have 20 people in the office - and 1 remote worker - your conference times should be at the preferred time of the 20.
My favorite exchange at the meeting couple of weeks ago (with a remote sales rep).
Meeting organizer: Just press the Start button and then type... Sales Rep: I don't see any Start button.
This went on for 5 minutes with 20 people in the meeting waiting around. Turned out, sales rep got a new Windows 8 laptop - no Start button.
Once when I was remote, I specifically arranged before the meeting with one coworker to set up a kibbutz session (kibbutz is an expect-script that works a lot like screen, and screen may well have worked, too) showing my terminal, and she was hooked up to the projector, so I could visually walk everyone through my code.
There are other issues with remote work, but honestly the company should spring for a decent phone. A few hundred bucks is more than enough, and your business surely has other needs for a decent speakerphone.
For instance, at a meeting 20 people have to wait for that one guy to get on the phone. Or now you have to pass the Klingon mic around so that the remote worker would hear properly. And god help you, if the remote folks are in a different time zone - now 90% of the people in the office get inconvenienced because someone is 3 hours ahead of us.
We have this very situation at work and it's very unhealthy and annoying. I've come to conclusion that on large teams it's either everyone is remote or no one is.
Hanselman (http://www.hanselman.com) has a good series on working remotely and making it work - he outlines the problems and how to solve them. At least he is trying - other folks simply do not.