The downside to Hetzner is their support is fairly brutal when it's outside their normal parameters: obvious hardware fails are replaced reasonably quickly (eg within an hour) - anything non-obvious and you'll batter your head against their support, or you'll have to pay extra to change hardware if you can't prove that it's their problem.
I had an EX4S and after a month I terminated it because it kept failing on a minimal Ubuntu install and they refused to do anything without proof or money, but have been very happy with a cheap 8GB dual core Athlon server I got through the Server Bidding process.
The time you spend doing sysadmin things could be spent writing code or otherwise growing the business.
Also, as you grow you aren't going to need to hire a bunch of sysadmins just to build, manage, and deploy more infrastructure. Saving even one $60,000 a year employee buys you what, $5,000 a month in infrastructure? That's quite a bit on Heroku + various addons.
How do you deal with hardware failure when you have a dedicated box at Hetzner? Specifically if/when something fails are there spares etc?
There is no one-size-fits-all or simple, formulaic solution to choosing cloud vs. bare metal. It's hugely dependent upon the organization and the application.
Just to play devil's advocate, it's not unknown for large VPS providers to have major issues. This is something you need to consider regardless of whether you're using dedicated or VPS.
I'm not talking though about a separate issue which is proper backup procedures. I mean if you have a server racked at Hetzner (or elsewhere) what is your "plan b" when there is a hardware failure.
In the case of a server that I just racked somewhere I will purchase a supply of the most logical parts to fail (fan, hard drive, board, power supply etc.) so that the parts are available and can be replaced quickly. I know that some providers take care of this for you so if the hardware fails (your hardware) you are back up and running quickly.
In the case of a VPS, by contrast, it can generally be assumed (nice if a way to verify this but I'm guessing there isn't) that they take have taken care of and planned for hardware issues and spares and have a strategy. Of course if they haven't you have a big problem. At least with your own hardware you can plan accordingly and insure a better outcome.
There are other issues as well. If the colo place is close to you do you keep the spare parts yourself or do you leave the spare parts with them? The answer depends on many factors (such as is there security where the parts are and who has access to them. If that is not the case better to keep the spares and drive over yourself with them even in the middle of the night).
I currently spend $100/month on 4 Linodes (3 x 512MB, 1 x 1GB). I love Linode -- efficient support, and their London datacentre has been utterly rock-solid for me for several years -- but I'm beginning to think that, for me, it's the worst of both worlds.
On the one hand, I could move all 4 servers to a dedicated Hetzner box (EX6 or EX6S) running Xen, for a small setup fee and similar monthly cost, and get 4 or 8GB ECC memory on each one. This has a slightly higher sysadmin burden (5 servers to administer instead of 4, slightly higher risk of disk failure), but not that much. And the move is relatively painless, because I can directly transfer the disk images with dd over SSH.
On the other, I could move the services to Heroku, probably pay a bit more, and essentially stop doing any sysadmin. This is superficially attractive... but moving a load of old things to Heroku isn't straightforward, and that probably rules this option out.