From what I understand, the phytoestrogens take the place of real mammalian estrogen, but our bodies don't process it like real estrogen, so its actually good for men who want less estrogen. It acts almost like an estrogen blocker.

Whereas drinking cow's milk, for example, is high in estrogen because its from it comes from a large female after giving birth.

Not contradictory, but working to different ends.

"You shouldn't worry about this because phytoestrogens aren't present"

"You shouldn't worry about this because phytoestrogens are beneficial"

Both can be true at once, and support the end argument, it's just not particularly helpful or harmful if both are true at the same time.