Many non-cyclists' perceptions of what is safe differ greatly from reality. For instance, passing can be dangerous, but intersections are where the vast majority of the risk are. And it also shows that a lot of dedicated bike infrastructure is designed to address perceptions of what's unsafe rather than what is actually unsafe. A lot of "protected" bike lanes in my view increase risk because they make the cyclists less visible at intersections.
I recommend looking at websites like https://bicyclesafe.com/ for a more realistic view of the risks of cycling and how to reduce them. My own style of riding makes the top 4 collisions there unlikely (I often "take the lane" as appropriate to improve how visible I am), though I do have issues with various types of right hooks. I'm not sure whether those can be avoided as I ride with hi-viz gear and try to be as visible as possible otherwise.
You're rolling the dice every time someone overtakes you. At least in a car I have some of the best engineering available dedicated to protecting me (crash test engineers are gods). Do you value your life so little?