I think if you look at race, sex, and disability discrimination, there's an implicit direction as well. The law is not particularly worried about companies refusing to hire people without disabilities.