Because it's extremely harmful to sex workers and many (perhaps a majority) of those who do it would prefer not to but lack the tools or support network to make a change.

What's the positive to society that justifies how frequently these women get robbed, assaulted, and raped?

"People routinely attack this group, QED we should ban this group" is odd logic. Lumber workers have the highest fatality rate of any profession in the US, at 1-in-1,000, should we also ban them? When queer people get attacked by homophobes, should we ban them from being queer?

Why is prostitution a different situation than those?

And how much of the violence against them is because they've been ostracized and routinely lack legal protections?

Is it really better for society that people who do prostitution to avoid starving to death... instead starve to death?

If you really believe people only do sex work because they lack the tools for something else, why not give them those tools instead of removing what little they already have? If your theory is correct, no one will decline these better tools; and anyone who does decline is reasonable evidence that your theory was wrong and sex work isn't actually intrinsically harmful.

Certainly, there are nations that legalize prostitution without falling in to anarchy and violence, and I don't just mean the Swedish model.