You seem to mix up binary with bimodal. Sex is indeed not binary, as illustrated by the many cases of "it's complicated". You can try to define a strong line, but the resulting rule will be quite arbitrary and mostly be based on your feelings. Accepting the distribution to be bimodal obviously still allows most of us to easily be categorized as either male or female. Same for gender roles, by the way, though with a bigger focus on behavior than biology.