I have moved dockerized apps to public cloud CF to on-prem CF. It all worked because the apps where stateless to begin with. Pivotal appropriating all the merit of easy migration of stateless apps is dishonest at best.
Stateless apps are easy to move. They can be moved from CF to Kubernetes to ECS to Heroku.
Now, let's talk about the state, because that's where the real problem and the lock-in might be?
State is hard, state has momentum. We do what everyone else does: we push it out into a separate parallel space. Heroku do that with their PostgreSQL and Kafka services. We do it with services. IBM, Amazon, Google, Microsoft all do it with services.
Cloud Foundry is agnostic to the services. More agnostic if you use BOSH. Less agnostic if you use a non-BOSH service (like RDS).
Heroku isn't. You're married to their PostgreSQL unless you want to build your own or switch to RDS.
Kubernetes has more of an emphasis on attaching and detaching volumes. I can do that with BOSH; in future it'll be an app-level feature too.
(edited to remove unnecessary grump)
In terms of quitting Cloud Foundry entirely, you have two ways out.
The first is that you used buildpacks to stage your apps. In that case you stop pushing sourcecode to CF and replace it with whatever build pipeline makes sense for the replacement platform.
The second is that you decided to push container images instead of sourcecode. In that case you push to the replacement platform instead.
That said, you'll need to recreate the other niceties: orchestration, routing, log collection, service injection and service brokers, self-healing infrastructure and a bunch of other bits and bobs.
All the IaaSes have APIs for fundamental components like compute, storage and network. BOSH provides a relatively uniform layer for them, which is how Cloud Foundry manages to get around to so many platforms.
Lots of customers want multi-cloud capability: they want to be able, relatively easily, to push their apps to a Cloud Foundry instance that's in a public IaaS or a private IaaS. They want to be able to choose which apps go where, or have the flexibility to keep baseload computing on-prem and spin up extra capacity in a public IaaS when necessary.
It also happens that lots of CIOs have painful lockins to commercial RDBMSes, and they don't want to repeat the experience. They want to avoid being locked into AWS, or Azure, or GCP, or vSphere, or even OpenStack.
CF is designed to achieve all of these. The official GCP writeup for Cloud Foundry[1] literally says "Developers will not be able to differentiate which infrastructure provider their applications are running in..." (can't say I completely agree, GCP's networking is pretty fast, which is I guess what happens when you have your own transoceanic fibre).
If I push an app to PCFDev -- a Cloud Foundry on my laptop -- it will run the same way on a Cloud Foundry running on AWS, GCP, Azure, vSphere, OpenStack and RackHD.
[1] https://cloud.google.com/solutions/cloud-foundry-on-gcp