I was debating disagreeing with the equivalence with alpine hiking. As I said in another comment, it's mostly about judgment when it comes to alpine hiking. Although perhaps that’s the case with diving as well. Of course, get more into mountaineering and there are a ton of factors like avalanches you can't fully protect against. (And there's of course some overlap with winter hiking in the mountains.)

ADDED: re: fitness. I've definitely had to take people down when I've been leading a trip and even had to turn a group around. But so long as you don't push on past where it's safe, it mostly just means people don't get up the mountain. (Of course, it can be a problem with canyons.)