If we're talking about relatively small workloads, and relatively stable traffic, then sure.

But I think for large workloads with unpredictable requirements, the capacity planning alone in a perm setup is a nightmare for most early-stage businesses. Spinning up an extra hundred instances in EC2 takes minutes - getting the same number of boxes installed a colocation facility takes weeks at best