It took about five years for the cultural zeitgeist of Germany to swing from extremely liberal to extremely fascist starting in the late 1920s. Sure, false flag and other political shenanigans hastened the process along - but it was the culture, or in your words, "the market", that managed to move rapidly to the right quite alright on its own.
Imagine if the nascent nationalist tendencies in the US and Europe really start ratcheting upward in a serious way over the next decade. Imagine that, at some point, "the market" judges pro-healthcare or pro-LGBT ideas to be harmful and best avoided. What then?
My point is that what you've done with your argument is handed Nazis all the moral authority they need to repress your ideas if their ideas become popular.
The argument for free speech is that we pay the small price of hearing vile ideas as insurance premiums against good ideas being censored if the pendulum ever swings the other way.
You're implying that the group of people - the subset of the German population - who were (allegedly) extremely liberal transformed into the group of people that were extremely fascist.
I don't have the time to research this, but it seems much more likely that these were largely - if not entirely - different population subsets, and that the latter arose in reaction to the former. It's not like the two groups had the same shared values with the new addition of "kill everybody that disagrees with me."
It isn’t limiting free speech by not wanting to listen to or follow people whose speech you disagree with and the argument that doing so is equivalent to how Nazis came to power is entirely historically inaccurate.
How can defenders of "free speech" say that it's ok for them to say whatever they want, but it's unfair for other people to respond to them? This is a tactic especially favored by the alt-right. They claim they're being censored and appeal to "free speech" the instant someone calls out poor behavior, and then turn around and organize massive harassment campaigns against people who say things they don't like.
The position is logically inconsistent, but they've found that just saying "free speech" makes people want to defend both of their actions. It's a wonderful exploit targeting people who have been ingrained to enthusiastically defend people saying "free speech", regardless of whether or not what they're actually doing resembles respecting the rights of as many people as possible to speak.
I'd like to point out one more thing. This isn't censorship. This is what the real free market of ideas looks like. You say things that people find vile, and the market judges your ideas to be harmful and best avoided. That's how it's supposed to work.