The conference gets to invite whoever they want. It's a private event, and they can set any conditions they choose for attendance. They can indeed write that policy and enforce it by revoking access. As far as I know the only US limits are when you're excluding a group legally protected from discrimination.