When the RFSL and Amnesty made their announcement to work for legalization there was a lot of media focus on the question. There was a lot of anger from people and articles about people leaving Amnesty and the Swedish branch making statement against the decision.

It was also not many months ago that V (the most left leaning party) suggested that the punishment for paying a sex-worker should be increased to similar levels as rape, with jail time as minimum.

We could say that V views are a fringe part of the feminist movement in Sweden, but I am not sure that is actually a correct assessment. As you say, few, if any, public figure dare to speak up with opposing views on the topic. It is hard to estimate popular opinion outside of political polls.