Swapping LLM models isn't hard, but if you build a production app or business process around it, how much time/effort is the testing to have confidence?

Which is easier when maintaining an LLM business process, swapping in the latest model or just leaving some old model alone and deferring upgrades?

Swapping is easy for ad hoc queries or version 1 but I think there's a big mess waiting to be handled.