> Your premise is that, rather than free association, every group of people owes a kind of due process to everyone else

They very well might. When you exercise your judgment in any way, you are liable for damages. Due process might protect you from that.