Even people who think that MJ is immoral or wrong may be against outright ban. It is not purpose of criminal justice to enforce virtue or morality per se, unless one believes in very strict interpretation of their religion (even religious police powers in Saudi Arabia were recently watered down). Victimless crimes will always wake some kind of unease in free-minded people.

P.S.: Politicians may not always act on wishes of silent majority. If the minority is strong and aggressive enough, they will be tempted to yield. Same dynamics as on Twitter, in principle; that is why ballot initiatives, where individual voters cannot be intimidated into voting in a certain way, are so important to democracy.