I'm the antithesis of an expert in this. If someone has better information, I'm sure you'll let me know. What I've picked up thus far as the consistent physical differences between male and female brains are:

1. Dominant synapse connection direction. Males typically have more front to back connections than side to side. This may explain why females typically are better at multi-tasking, and males are typically more focused.

2. Testosterone in the womb enlarges the 3d manipulation part of the brain.

3. Testosterone in the womb has some connection to sexuality. In particular, women typically show signs of arousal towards sex without any filter (ie, animals, same gender, w/e). Typical males will show no signs of arousal to certain things (typically other males). Also, twins boys have a significantly greater likelihood of homosexuality (less testosterone to go around).

There's plenty of other differences I'm sure, and many of these may not be true. But this is what I've noted thus far.