> Its because its not allowed, we either have a list of protected groups - or a general rule that forbids banning anyone because of their 'group'.

It's the former. There is no law, at least in the US, that forbids banning people from private property for group membership in general.