From what I read on the topic, it's much more complicated. People have plenty of typical-female or typical-male parts of brain slushed around, generally people tend to have the most of them that correspond to their gender (IIRC you can guess gender/sex correctly about 80 or 90% of the time by looking at the brain).
It may contribute along with other factors for the cases where the brain does express gender dysphoria.
So really, I don't personally believe a brain can be cleanly male or cleanly female. Rather, if specific parts are formed differently than usual, then they respond differently to external stimuli. This can result in a wide range of sex-atypical behaviour ranging from mere homosexuality to finding one's genitals revolting. How you interpret that objective phenomenon is really up to the individual.
Though I also do agree that gender dysphoria can have many different expressions, once you drop gender you get into a whole new world of stuffs where people hate various things about themselves (I've met people with self-described "whole body dysphoria" who are generally okay with their gender and simply hate the physical existence of their body)