You're right! Which is why such a case is going to the United States Supreme Court, because it's not a settled question.

But let's say, refusing to bake a cake for a black couple?

There exist groups in the United States we consider protected classes, and the state or federal government passes laws to protect these classes, or court precedent protects them from discrimination.