The way it works in safety critical software safety analysis in my experience is that you have a hazard analysis/failure modes effects analysis that factors in severity x probability (and sometimes a detectibility measure)

So if you identify a failure mode that contributes to a catastrophic hazard for instance, you better build your system to drive the probability down. The resultant severity x probability score you end with has to fall within the risk parameters deemed acceptable by management/safety