Especially given that many orgs use managed or cloud specific solutions that have no 1:1 mapping between vendors
As someone who worked on a similar project recently, I'm getting that idea that you obviously don't know what you're talking about.
You need to plan ahead, sure. Portability was one of my main concerns (including the possibility to go self-hosted). But it's definitely not impossible, nor too hard to do.
And Kubernetes? It goes way beyond containers, you can replace many cloud provider-specific resources with Kubernetes resources. What alternative gives you that?
This will almost certainly lead to a prompt rebuild, to better accommodate new model idiosyncrasies.
If you are unlucky- your use case may be one where Evals require human review.
Unless you are YOLOing it without evals. In that case this is relevant.
Margin is a function of stickiness/cost of switching (among other things).
I suspect eventually we will enter a world where migrating cloud providers is mostly a click of a button, but we're a long ways off from that. Requires vendor agnostic and portable apis/containers/WASM runtimes everywhere.
Swapping an LLM, at least in the current state, is about as close to updating a pointer as you can get