I don't think decriminalization of hard drugs works without an ample amount of long term money set aside for treatment. Plus money for additional law enforcement to aggressively enforce existing property, violence and public intoxication laws. People should not have to endure a slipping down life because others choose to do drugs.
As far as the why people do drugs, I'm not sure we need that much study. It starts out fun, then it turns into something to numb the pain, then its something that is needed. I've done a myriad of drugs in my time, thankfully I never passed the fun phase. I don't presume to know what drove people to the 'numb the pain' phase and I do feel for them. I am not willing to upend my life or endanger my kids to find out. There is also a limit, how much money should society be expected to spend to constantly on an individual who keeps going through treatment and keeps going back to using? With that said, I do think the number is above zero.
My patience fails though at those that endanger others and commit crimes to further their addiction.
I'm just someone who thinks drug use happens for a reason and doesn't get resolved with sheer self discipline. It gets resolved when the reason for using gets resolved.
-- Albert Einstein
In a later comment you ask ...how should I provide help & compassion?
I would say one-on-one as an individual walking the streets, you probably shouldn't, especially if you have children in tow.
We need to address our nationwide affordable housing crisis. California is disproportionately impacted and takes a lot of the blame for it, but I think they are basically the dumping ground and whipping boy for a national failure.
Decriminalization is helpful in part because if you criminalize use of the drug, it makes it harder to seek treatment. People who expect to go to jail for trying to get the help they need are more likely to not come forward and to continue to try to fix their problems themselves.
We need a better understanding of why people do drugs, what the underlying root causes are. It's hard to solve a problem without knowing why it exists. Given our long track record of not getting results, I think we have room for improvement in how we frame this problem.