Other than that pesky nebulous western value of "Freedom", sure, I guess you have no reason to be able to make that kind of business decision. Compelled speech or compelled production of value come in many disguises, and are hallmarks of oppressive and dangerous regimes. It's disingenuous to suggest that starting down that path would be innocent.

But these things are already happening, such as forcing religious bakers to bake cakes for couples they don’t agree with

Of course, because the reasoning wasn't 'you are a jerk' or 'we don't want to make that type of cake' or 'we're too busy right now' but rather 'you are gay'. Sexual orientation is a protected class and acting like a jerk on the internet isn't.

No one is forcing all businesses to do serve all customers everywhere always. We are saying you can't discriminate on the basis of marginalized groups.