That's not how the law treats, say, selling houses: it's not a homeowner's right to sell their house to whomever they feel like selling for and not whomever they don't, if that feeling involves discrimination on race.

I think this discussion is likely to be more productive if folks who hold the belief you're holding also state what their agreement is with existing laws restricting how people can participate in the market: if you also disagree with housing anti-discrimination laws, that's fine, but it'll be easier to understand your position if you say you're advocating for those being unjust too. If you don't, I'm curious where you draw the distinction.