The fact that how substances are regulated is irregular and inconsistent is a separate issue from whether or not it's ok for a society to regulate some substances.
The bar for "affects others" is extremely low in society.
Unless there's a very clear study, that a certain substance will cause an average human being to be aggressive, belligerent and violent - there's hardly a case for "affects others".
All drug use has risks, including cannabis. But it's been so demonized in the past there's a natural reaction to sweep all concerns away because almost all of them are garbage. Reasonable regulation would be that it's not sold to minors, and that its active ingredients be accurately noted, and that the transaction is taxed appropriately.
It should be legal to sell fentanyl but not as an adulterant. Most of these opioid deaths are due to fentanyl and if they had access to regulated drugs those deaths would be greatly diminished.
My brother died of a heroin overdose many years ago, and I think he might be alive today if the drug he was abusing was at least calibrated for dosage. Maybe not. But it being illegal did not protect him or any of the others lost in the war on drugs.
This is well documented, the above link is one of many.