It's complicated in the US because the person at fault is also on the hook for medical costs of any injured parties. The car insurance you get is mostly about taking care of that, not the property damage.
FWIW I didn't downvote you. Your post is borne of ignorance of how it works in the US, which is actually enviable because it means your healthcare system isn't an absolute wreck :-)
In the "no mandatory insurance" scenario, you're taking the risk that someone else may damage your car. So don't buy a car you can't afford to replace.