You don't live in a vacuum or a state with a population of one. If your actions have consequences - and most do, even if it's only through the socialized pooling effect of insurance - then there is a moral argument for society to regulate them.

Personally I'm more of a fan of utilitarian thinking that takes into account second and third order effects. It's not enough to regulate X because X is bad; you need to look at what X is currently displacing, black market effects, underlying demand. But it's just silly to deny the moral legitimacy of regulation in itself - we all live in a society and it's childishly selfish to think you can just do as you please. It's prohibitively expensive to capture all the externalities of our actions, so we regulate instead.