Setting a lower resolution format via "ytdl-format=..." in the mpv.conf file may resolve that. Personally I also use "pause=yes" and find that the few extra seconds it has to cache also helps.