You're ignoring the history of prohibition (of alcohol and drugs) and the 'war on drugs'. Not to mention the 70's, the connection between recreational drugs and 'hippies' (a subversive anti-war class). Not to mention the fact that up to now the 'war on drugs' has also been used to restrict the rights of certain social classes (it's been well documented how the laws are designed to criminalize certain classes/ethnicities disproportionately).
You're just rationalizing.