I think the burden for imprisonment and compelling people to stay in prison is on the punishers.

That should not be confused with the crimes being wrong, I'm talking about ones desire not to be imprisoned being a natural desire and acting on it not being wrong by itself.