Now a population of mixed male-type and hermaphoditic-type individuals by definition can only skew male-type (in the sense that the average individual over the entire population somewhat prefers fathering over mothering). But each offspring has one male and one female parent; i.e. the population of parents does not skew in either direction. Run the math and you'll find that, counting hermaphodites as 1/2 in each category, there are more offspring per female-type than per male-type: those hermaphodites who prefer mothering are now the sexually fittest individuals in the group. And much like in the case with males above, a genetic preference towards mothering will catch on among the hermaphoditic population and begin skewing a subpopulation ever more female-like.
Eventually the male-type and female-type sexes fully diverge from each other, and the initial population of hermaphodites get out competed by the sexual specialists and fall solely into the fossil record.