> Yet people aren't generally freaked out when people drink alcohol.

I think you might be discarding your priors here. Alcohol use is extremely common, so even relatively uncommon negative effects will happen a lot. The important number to compare is how many people would see equivalently bad effects if the same number of people used <other substance X> as drank alcohol.