This seems a little like arguing that I know several people who died in car accidents, and none in spaceships. One of these is much more dangerous, but almost everyone I know drives places, and has a drink or two, every week.

Maybe that's too strong, I'm not really sure how much more dangerous cocaine is. But even if it were somehow comparable, maybe we should be scared by what the arrival of alcohol did to peoples who hadn't had much of it -- this was true of many native american groups, IIRC. And many others worldwide.