The difference in the swiss city of Geneva at the central station in the late 80s and early 90s is hell and heaven. While it does not solve all problems the methadone heroin replacement system does solve the vast majority of negative issues. Petty crime and robbery dropped significantly, whores for drugs disappeared, needle reuse is non existent. And in the 80s I was warned with reason to look out for drug needles and not touch them. This is not a worry anymore, either in the Netherlands or Switzerland.
Both Amsterdam and Geneva suffer these days from organized street crime. But the risk of being attacked by a junky for your wallet is minimal. Organized street crime prefers to steal from you without you noticing and are quite skillful. Junkies needing money for a hit are more like dogs with rabies. The remaining junkies are rather fragile and slow, and not dangerous as long as they get their regular hit. Which they do in a safe and controlled environment.
Having been to San Fransisco and Amsterdam these are no longer comparable at all. 25 years ago I can believe but 15 years, no.
I think many people have issue accepting that drug addicts are part of society. Criminalize them, and the whole underworld evolves but it won't affect their numbers much. Put death sentence on drug use, and there will still be healthy population of addicts.
So with your 'drawing line' approach and misleaded comparison to sex trade, you just give ammo fodder for far right-wing populist politics, polarizing society and law enforcement, increasing crime and making it more organized, while achieving very little positive impact. Something like famous US screw-up called War on drugs.