Is pleasing Google now the only reason to obey the HTTP spec?
For example, I had a hard time telling people that they should use a proper HTTP redirect from their domain "example.com" to "www.example.com" (or vice versa), instead of serving the content on both domains. All arguments about standards, best practice etc. were unconvincing. But since Google startet to punish "duplicate content", I never had problems convincing people again.
It's simply good practice to get the HTTP status codes correct in the ways you outline.
If I'm being honest, it's off putting to watch community-wide apologist response to what would normally be outrage of poor execution.
But you nail it. Had Gmail, Reddit or any other unpaid service gone down like this, the uproar would be heard from another galaxy.
In what universe? If email were to stop working in most major corporations that I've worked in for the last 8+ years, the company would basically come to a halt.
Email, for many, many companies is the message/workflow bus, and if it stops - communication comes to a halt.
It is, after electricity, and the network, the one essential function in a company in 2013.
Telephones, Photocopiers, Printers can all cease functioning , with little impact in most technologies companies - but not email.
I don't know where you keep coming up with that - Email is relatively easy to make highly available, and an Exchange Server in 2014, configured with even a modicum of skill, will likely keep running smoothly for the next 10 years. It's as close to 5 9s, of availability that you can get on a software system.
"There has been no big improvements to the reliability email the last 40 years. "
That's just silly. Take 5 minutes to read through http://en.wikipedia.org/wiki/Email and you'll see what improvements have been made in the last 40 years.
You also mention competance, which is available in variable quantities. You might be able to keep Exchange 2014 running solidly, but I've seen people doing scary things with MS SBS Server 2000 (and the exchange that comes with that).
Or data -- they tossed a day or so of submissions and comments when restoring from backup, AFAICT. I'm not saying that's a problem; 4chan loses data and works just fine.
1. If you add the "max-stale" header to your content you can tell your CDN to hold on to things longer in the event of an outage. What happens is that the CDN will look at the cache time you set and check in as normal when that expires, but if it gets back no response or a 503 code (server maintenance status code- if you work with CDNs this is your friend!) it will continue serving the stale content for as long as you tell it.
2. Lets say you're beyond that, or your site is too dynamic. Instead of setting up an error page that responds to everything, setup a redirect with the 302 status code (so crawlers and browsers know it's a temporary thing). Point that redirect at an error page and you're golden. The best part is these types of requests use minimal resources.
What I do is keep a "maintenance" host up at all times that responds to a few domains. It responds to all requests with a redirect that points to an error page that issues the 503 maintenance code. Whenever there's an issue I just point things at that and stop caring while I deal with the problem. I've seen webservers go down for hours without people realizing anything was up, although with dynamic stuff everything is obviously limited. The other benefit to this system is that it makes planned maintenance a hell of a lot easier too.
Oh, another thought- you can use a service like Dyn (their enterprise DNS) or Edgecast to do "failover dns". Basically, if their monitoring systems notice an issue they just change your DNS records for you to point at that maintenance domain. You can also trigger it manually for planned things.