One advantage of "Heroku's" nickel and diming is that the money is actually being split with other companies in most cases, and those companies core competency is providing exactly that service, whether that be a Redis server or email delivery.
You do pay more than just throwing another service on a machine you're already running, but you've got a fairly good chance of it being massively optimised.
The bottom line is that dedicated hardware is cheaper because it comes with less. If you aren't making use of those value added services, then by all means go dedicated. If you know you don't know what you don't know and you want experienced support to take responsibility for a larger percentage of the stack, go Heroku or Engine Yard. If you have specific overriding requirements, let that guide your decision to a platform. There's no one size fits all solution, and cost is not the primary difference.