My quote is "everyone leaves Heroku eventually". Either you hit something that Heroku won't let you do (because of their tightly bound model) or you are burning tens of thousands a month on what would cost 20% on AWS/GKE, and you do it for cost savings (people are nontrivially expensive, of course)/ownership.

Everyone leaves Heroku eventually. :D

Most sites never hit a scale big enough that they would need to leave

You can't easily scale with Heroku. Scaling, almost by its very nature, requires tight control over your infrastructure.

When you scale, you need to make your infrastructure fit your app. Not the other way around.