True, a male can develop with malformed testes or a woman with non-functioning ovaries, but can we still understand the difference between the sexes. For gonochoric species like humans it's the whole body plan that develops separately for each reproductive role.

It's not just hormone levels measured at any one point in time, but the result of a long and specialised developmental process. For instance, females and males end up having different skeletal structure which we can observe as accommodating childbirth in the former.