Yes, if everyone agreed that your definition of fairness were the right one (which I personally do) then there'd be no problem. But in practice there will be people who want to use other ones (in some cases because they want to be able to say it's discriminatory).