There is a huge confirmation bias. You only hear about the horror stories, while people who are perfectly fine are afraid to admit so much, because of social stigmas and prejudices. Hell most companies require a drug screening as a condition of employment. People who smoke pot aren't going to come out and mention it at the water cooler for fear of retribution.

I was talking with a co-worker about US laws in general and he mentioned something I'd never thought of. Many of the laws we pass, speed limits, alcohol tax, prohibition, .08 legal driving limit, etc are designed for the lowest common denominator. Meaning, we pass laws based on the worst case scenario. It seems like it's illegal to do just about anything these days.