It's very hard to draw a line between things that do and don't harm others. We should run an experiment among your friends and family, co-workers, measure any effects on your health and associated costs covered by the tax payer etc.

Don't get me wrong, I smoked pot daily more than half of my life. I know I'm a better person without it and after quitting my long term girlfriend said the changes were incredible. That's not an argument for criminalising this plant, but the argument really isn't as simple as a one-liner.