The value is that when support staff has real phone numbers available to them, they tend to dial historically responsive individuals directly in order to get a problem resolved, thus creating a negative feedback loop -- if you ignore notifications and phone calls, you get called less by support staff in the future.
Having a floating number -- especially if we could get statistics on who answers and who always ignores them, and if the number could "call up the chain" automatically when nobody answers, would be a useful tool to solve this issue.
Of course, the preference is that human staff doesn't need to call anyone, but it still happens.
That's an interesting idea. We've actually thought a bit about adding phone-based triggering to PagerDuty (via a 1-800 number + access code). The idea was to make PagerDuty useful to non-IT businesses like plumbers that also have the concept of out-of-hours on-call duty. From your comment, though, it sounds like this kind of feature would be pretty useful even in the IT world.
This encourages the following:
- Closer correlation between business robustness requirements and software implementation.
- Adoption of more robust design or methodologies when required by the business. If all that is required is an automated restart, then automatically restart the software. If the same class of bugs causes regular failure, adopt a strategy for avoiding that class of bugs.
- A realistic platform for negotiating external support. If engineering staff is unable to produce software sufficiently robust as to support the business requirements, then we must make a business decision as to decide whether maintaining additional operations staff is cheaper -- in the short and long-term -- than correcting the engineering issues.
In my experience, software that fails regularly enough to cost significant engineering resources in responding to those failures is generally broken software. The goal is to not let software get to that point, and to correct it quickly if it does.
It's very easy for your Operations budget to unnecessarily balloon under the load of supporting failure-prone software; engineering has every incentive to externalize the costs of their implementation decisions, while operations has every incentive to increase their headcount and budget by supporting those failure-prone systems.
As an aside, I find the best way to avoid regular failures and decrease the necessity for a large operations staff is to put the individuals responsible for building the system on-call for when it fails. Your operations staff is woken up when a server crashes or a hard drive fails, and your engineers get woken up when their code crashes in the middle of the night.
If you don't do this, the costs of writing poor production code have to be levied across departments by management, rather than avoiding externalities entirely and letting engineers and operations deal with the direct impact of their implementation choices.
Of course, this is ultimately a wash if you don't also institute development methodologies to help reduce the number of production-impacting bugs, rather than simply relying on engineer's reactive fixes to one-off issues.