In fact the law does forbid certain forms of discrimination by "we the people." You cannot deny someone a job on the basis of age, gender, sexual orientation, race, religion, etc. Redlining is illegal, discrimination in rentals is illegal (as the previous President discovered), etc.