While I agree with the premise that societal restrictions on behavior might be improved if they involve tests for harm, the particular constraints you disagree with exist precisely to prevent material harm.

Anti-discrimination laws are built on a history of groups being excluded from access to housing, loans, schooling, contracts, etc. I don't think it is hard to see that depriving certain groups of access to the instruments of life, liberty, and the pursuit of happiness causes material harm.