Exactly the same thing happens with mpv if you shift the audio manually. I experienced that recently. After shifting, whenever I go forward or backward it gets out of sync + the performance decreases ( without the shift going forward or backwards is instant)