There are also a whole bunch we've run into that were largely isolated until recently in Australia and the pacific. How many of them are matriarchal?
Zero.
Genomic studies are increasingly suggesting an almighty y-bottleneck 20ka ago. One of the things this is consistent with would be a matriarchical society, as females would control reproduction. Couple this with "Venus" figurines scattered across Europe, evidence for societal upheaval ~15ka ago, and a possible and plausible explanation becomes that females were running the show for much of history, being the ones that can bear young (c.f. Animal hierarchies), and males took control recently, possibly in response to encountering other humans (neanderthalis).
Anyway, it's all very patchy and speculative right now, but further genomic studies and archaeology probably can answer this one.
You are absolutely right however that it plays both ways - we don't actually yet know enough - but something happened.
We don't know that, and probably will never know. The problem is that those societies left little that we can use to learn about them.
Our main source of information of our hunter-gatherer ancestors is often modern hunter-gatherers, but they are very likely to be different than the ancient ones. Two major differences are (1) the modern ones are in contact with modern societies, and (2) the land they live on is not random, it's land agriculturalists didn't decide to occupy, and has some specific characteristics (mainly that it is poor).
It's also very possible that there isn't a single truth. Hunter-gatherers existed for a long time across a great many locations. They may have had a large variety of cultures.
In support of that, we do have evidence of the opposite of your claim, too: hunter-gatherers that fight wars to capture women, for example.