You just shrugged off opium dens, or moonshining, or whatever substances could be had. The demon liquor and marijuana and heroin were all well known.

But, people didn't care about the poor. Whether or not they do now, they do collect data about them. We don't necessarily have great understanding about prevalence of who used them how often, so we shouldn't pretend there was a significant difference unless there's a reason to. Nostalgia or rose-colored glasses or pretences of a better time aren't valid reasons to break from expectation.