I think that's part of the case part of the time, but it's definitely not the whole thing. Other prongs include purity (drugs are a corrupting influence) and tribalism/classism (drug use marks you as an inferior and unwelcome kind of person).