If we both wear a mask then we pose very little danger to each other. We pose a far greater danger to others driving a car and having a collision

In France for example, there has been ~35'000 deaths because of COVID-19 in ~6 months of 2020, while there has been ~3'500 deaths on the roads in ~12 months of 2019.

Your "if we both" is a very big if => without making it mandatory, people don't wear them and ill people can propagate the virus to many, making COVID propagation possibly more dangerous than a car accident they could create.

Making mask usage mandatory and actually policing it would be a much lighter touch and the evidence from Japan suggests the virus can be kept under control without lockdowns. If you take the numbers in Japan in your example, driving is more dangerous