Any AI is is non-deterministic by nature, this is more about statistics, but not dedicated test cases. So, even if you swich to the model which is better in metric results you still cannot guarantee it will be better in your specific use case or do the same. What you can probably do is to automate your important test scenarious and run each every time you change model or system prompt

I've been building a tool to do this - build a dataset based on tasks from your repo, then A/B test the agent with whatever change you're making to determine the impact prior to actually shipping it. If you want to check it out - stet.sh