http://www.zeus.com/products/traffic-manager
http://www.zeus.com/products/global-load-balancer
These are smart, easy to use and well supported. They're also to be found inside big cloud offerings like AWS and Joyent.
(pseudo code) if (curl http://ip of main site) fails then copy alternate zone file to bind dir service named reload fi
You get the idea... F5 Networks is really just a fancy DNS server running on a BSD based OS on an x86 appliance. Zeus, which another commentor mentioned, has an AWS version and will let you run it on your own hardware if you like.
I'd love to see some open source competition for this space, or even low price competition.
Yes, that's exactly what I meant. Running our own failover system is not just expensive, but time-consuming - just another thing to do when you are trying to scale and time is already short within a small team.
They are expensive, but this is how large companies like Yahoo keep close to 100% uptime.
Explain to the customer that even with a hot site, the failover can take a few minutes. Also, some ISPs don't honor TTL and cache DNS queries for longer than they should. The Internet isn't perfect, and usually each extra 9 you add is around 4x more cost.