You may be thinking of species like the white-throated sparrow. These have two morphs with distinct behaviours which lead to there being four mating combinations. Still two sexes though.