Do you have the same attitudes towards helmets whether you're walking on the street or riding a motorcycle? I'm assuming not, because you understand they are different risk profiles.
When risk profiles differ, so do the mitigation strategies. One of the ways spaceflight manages different risk profiles is by implementing a risk-based approach. For example, NASA classifies their software risk based on probability and severity. When risk is high, they leverage more requirements and more oversight as a way of mitigating the risk to an acceptable level.
What you seem to be insinuating is we should be willing to accept more risk. While I agree in principle, the hard part is getting stakeholders (from astronauts to contractors, administrators, politicians, and taxpayers) to agree. They get a say, too.