> To be clear, in my example, polyamorous people are only a problem for society because that society has decided that polyamorous people are a problem

Any problem for the society is like that. You can say robberies are a problem only because society decided it is. That's how society works - there's no some external source where it could be checked, so the implication that it's arbitrary baseless decision because it is social is wrong - all rules of thd society are such, so if you want to have any rules at all, those would be the ones you'd have. It's impossible to have a society where everyone derives their morals from the physical laws anew all the time.