I agree, but my reasoning is different. The simple fact is that not everyone knows which people in their circles could possibly supply drugs. So for the user in their mid-40's that would like to take lsd twice a year now that the kids are off to college, this is a hurdle.
With legalisation, said couple can go to a store and get them, even with all the rest of the stuff still in place - social stigma, financial costs, and even sketchy providers (though that is much less likely).
Also when people argue for legal drugs the vast majority are arguing for end-users to have the ability to purchase drugs through a regulated system, not the ability to air drone a pound of heroin to your house at a moments notice. The two most important regulations in my mind should be age restriction(to stop children) and a waiting period(like 3 months to stop impulse buys).
*Knew 4-6 people over my life who had their lives ruined through hard drug use, and no-one who's life was ruined through the direct legal consequences of hard drug use.
Some factors are completely negligible for some users. For example, individuals coming from a high income bracket rarely face significant legal costs due to their drug use. However, the story is much different for those from lower income brackets. None of the people I know that use drugs have faced any legal problems, but people are detained for carrying drugs. Just because I'm not witness to it, does not mean that cost does not exist for some non-negligible part of the population.
In another vein, for some people the legal repercussions are an issue due to their personalities or circumstances (drug testing at work).
I was merely saying that you could want the drugs, but there are still some strong forces (that pull on people differently) that make this decision far from frictionless.
Even with drugs that are pretty available like alcohol, alcoholics have to curb their desire and usage based on many external constraints. And there is definitely no alcohol drought, but there's also no permissive free for all for consumption.