We carve out those special exceptions for reasons of striving to create a fairer world for everyone: people in this country have historically been (and contemporarily are) discriminated against for race, gender, sexual orientation. The discrimination, both from people personally and encoded into laws and large-scale businesses processes, makes society unfair for those groups, e.g. redlining put people of color at a significant disadvantage for home-buying and building intergenerational wealth for their communities. Those are also things people are born with. Discriminating against people for how they're born, individually and systematically, runs counter to creating a fair, just, equal society.

Additionally, the freedom argument in the Jim Crow era, "let racist businesses remain racist and eventually they'll be replaced by non-racist businesses" was tried out for a hundred years, and the status quo did not change for people of color. It took the civil rights era and outright banning of discriminatory behavior by businesses for the status quo to change for the better/fairer/more equal. I don't think it would be a good idea to return to that older state of affairs, even if it means business owners must serve people of color, gay people, whatever, because, from my reading of history, that experiment failed.