A better argument would be that every time brain differences were found, they turn out to be irrelevant. However, I'm not sure even that is true, considering some colleges are experimenting with mental rotation training to help female engineering students, with good results: https://www.kqed.org/mindshift/43802/can-teaching-spatial-sk... That example also shows identifying gender differences is not a bad thing; it allows coming up with ways to compensate.
But the plasticity means that cultural practises are likely to have an effect on brains, in the same way that London Cabbies famously have over developed hippocampus with more nerve cells than they had before due to the requirements of their jobs.
Finding brain differences does not, therefore, mean that the differences are genetic or directly gender related, but could be culturally generated through different roles being adopted for social reasons.
https://en.wikipedia.org/wiki/Human_body_weight#Average_weig...
It seems pretty clear that absolute brain mass is not that important, but the ratio of brain to body mass, is. For example, whales famously have (much) larger brains than humans, but are not (considered) as intelligent as humans.
https://ofwhale.com/how-big-is-a-whales-brain/
The brain of the sperm whale is the largest, five times heavier than a human’s. The adult sperm whale brain is 8,000 cubic centimeters weighing about 8 kg (18 lb), while ours is about 1300 cubic centimeters. A human brain can weigh 1.5 kg in adulthood.
Note again: sperm whales have five times more brains than humans.
As the other posters have said, size doesn't seem to correlate to anything. More than that I don't know.
That aside the point is there would be less nerve input for the brain to process from five grams of fat, five grams of skin, or say an organ like the eye. Let alone any other other details it must acount or provide for. Further complicating things is neural plasticity - if I recall correctly "optical" areas of the born blind tend to be repurposed for other senses as opposed to say mathematical ability.
TL;DR: There is evidence but certainly not good enough to tell us anything about sex differences in humans.
https://en.wikipedia.org/wiki/Neuroscience_of_sex_difference...