For extremely low latency audio, you simply convert most jitter into loss (and the rest into added delay). How annoying the loss depends on the quality of the loss concealment.

If your conferencing is 'slowing down' and 'speeding up' then it's simply not a realtime communications medium.