We don't allow people to risk their own lives. We force people by law to wear motorcycle helmets and seatbelts (ignoring a very few number of states). That doesn't at all improve the chances for other people to survive. You aren't considering the fact that a bicyclist causes injuries to people who aren't them. There are plenty of scenarios for this, cars swerving into incoming traffic, into a utility pole into pedestrians to avoid a cyclist. Way more people die from swerving to avoid animals than from hitting animals; even hitting a deer isn't likely to kill you in a modern automobile, but swerving into a tree to avoid one will.
Except when they don't, due to mobility problems or having pets with them or other reasons. Expecting anyone to jump out of your way when you come barreling down the sidewalk is going to hurt someone.
A mistake on a bike is going to almost always be a life-or-death situation as just falling onto the road puts them at extremely high risk for injury. The cost of errors made by car drivers has a huge impact on the morbidity of cyclists, while the opposite doesn't hold true. Even if cyclists were an order of magnitude more dangerous than car drivers, their poor decisions would have negligible impact on car drivers.