Only when these rules come into conflict. People can have different preferences, include preferences about with whom to associate or not. Allowing for that is a necessity for living in a free society.