IMO - it's not the pleasure itself that is stigmatized, but the side-effects. Side effects including deteriorating health of the user, as well as the occasional resorting to immoral or even criminal means to feed the habit.

There's also a whole dog-whistle aspect to the 'war on drugs' (e.g. massive differences in sentences on crack vs. cocaine offences)