Sounds like your conferencing system has large jitter buffers. If your internet is stable, you should be able to reduce the buffer.

To test your network, go to https://www.dslreports.com/speedtest and see if you have significant jitter or bufferbloat. If your rating is an A, then your good to go, otherwise its time to clean up your local network :P