The restarting is independent from the games performance. Battle times actually eat into the time you have after the battle to update your team, and at some point restarting the game is faster than watching the battle unfold.