It's not as doom and gloom as you think. I've worked a few companies that colocated at a datacenters, bought some refurbished hardware slapped Xen and Proxmox on it and grew their business to $50mil ARR. It's much much cheaper than you think and avoiding large AWS bills was the primary motivation for doing it.

The budget for one company I worked at's infra -- 4 full racks in two datacenters with power, network, and a dedicated site-to-site between them was $200k over five years plus $3k/mo ongoing costs. The upfront cost was tiny tiny -- they started with a router, two switches and 4 used supermicro blades. That lasted them the first 4 years of their startup. It took two FTEs to manage the full infra and provide a PaaS style thing to the developers.