The benefit of AWS-style comes in at
1) Flexibile workflows (I need the grunt on mondays, but not the rest of the week)
2) Tiny workflows (I just want a couple of $5/month VMs in different cities and some DNS to host a couple of toys)
Now if you have the $8/hour self drive, that's great if you use it once a year on a 10 hour trip to a holiday destination. It's awful if you use it for 2 hours a day.
I do save the few staff hours needed on physical maintenance of racking, cabling and replacing hardware. But at typical startup scale that's not much work. In the past I've done it myself on the side. As the company gets larger it takes dedicated staff, but at that point the comparable AWS bill is so large I could pay their full year salary on a single month of AWS bills.
AWS have managed services for a lot of different things, which save a lot of time on maintenance. Object storage? Message queue? Any type of database? ML? etc. etc. etc.
And of course, there's the scalability. With on-prem it's a careful balancing act of buying enough capacity to have space to expand but not too much to waste it, and you have to regularly keep an eye out lest you hit a limit with multi-week lead times ( e.g. you're out of storage, and buying a new shelf can take weeks to be usable).
Physical colocation is nearly always cheaper than AWS for most use cases.
Exceptions are when your usage is so tiny that it doesn't need even a single full-time server, and, when your usage is so erratic that provisioning for the spikes doesn't make sense.
For everyone else, you're paying a very large premium for AWS over what it'd cost to buy and operate some servers.