That's right, but the negatives are presently having a large impact on society. There is a balance to be struck, but it's something that viewing the raw numbers can obscure.

Non-domestic terrorism seems like a good counter-example, where we have gone too far in an attempt to eliminate it: a huge financial and societal undertaking (e.g., the TSA, the justified mass-surveillance, the increase in public fear) for relatively few deaths.