We could start by not prosecuting done crimes. Why do you go to jail for possessing marijuana, prostitution, crossed a border without permission, or because you are awaiting trial? (Which is why most people are in jail.)

We start by taking these hundreds of thousands of people who really shouldn't be in jail or prison at all and ask, "what programs would help them from not doing this again? What's the root cause we can address?" And we start implementing social programs that will help support, educate, and reform folks.

Some combination of:

1) Remove a lot of drug crimes from the books and clear convictions. Many of these shouldn't be punishable at all.

2) invest in housing, food, and mental health support to catch folks before they turn to low level theft. Give them to people convicted as well. If you change their situations, many will never need to commit crimes again.

3) Levy fines, garnish wages while providing support. Restrict movement if needed by requiring folks stay in contact with a parole officer or mental health support professional.

Etc

Most "criminals" are folks who are in positions where we left them few choices as a society.

Most of what you're saying is that we should select prisoners better and avoid creating more criminals as a society. The principle I agree. However this part is completely non-sequitur, as the point of contention is whether there's something better than prison to deal with after-the-fact criminals. The only alternative you give is a very soft form of restricting movement, which is devoid of reality for many forms of crime from petty theft and embezzlement to rape and murder.

The goal isn't punishment, it's rehabilitation.