If the server hosting your EC2 instance fails, you just bring up a replacement and continue on your way. If you are on a traditional VPS, you either have to bring up a new instance from backup, or wait for the provider to get things running again, but at least they have some strong motivation to since multiple customers are down. If you have a rented or coloed server, you have to wait for someone to fix it, with shorter guaranteed turn-around times costing you more.
You might be able to make the case that you could deploy an app on a single EC2 instance because in the event of failure, your downtime probably runs much less than an hour. Downtime due to a hardware problem on a rented or coloed server is likely to be much more than that. Maybe that tradeoff is acceptable to keep the costs down, but it is a tradeoff. If you aren't willing to make that tradeoff, your cheaper rented or coloed server is actually two servers, with a rough doubling in costs.
Do you get the bandwidth cheaper elsewhere? Perhaps but probably not significantly.
Do you get storage with similar flexibility and scalability cheaper elsewhere? Highly unlikely.
Oh and housing, power and staff aren't free either. Neither are the premium support contracts with your various suppliers to get same-day replacements for failed hardware.
Once you factor everything in you'll realize that EC2 is not nearly as expensive as it may seem at a glance.
That myth keeps popping up but is simply not true in many situations. There is a wide corridor of use-cases in the mid-range where EC2 is indeed more cost effective than rented or owned servers. Sometimes dramatically so.
The number of factors involved is obviously quite large, but lets just consider the baseline cost of having a "large" instance running for 3 years straight: $10386 (or as low as $4510 if you commit by reservation).
You'll be hard-pressed to match that figure for a rented or bought server, simply because EC2 is a 50.000 server gig and likely yields slightly more economy-of-scale savings than your setup ever could.