It almost seems like there's a tradeoff between the two "brains"
This would explain why pandas, koalas, and sloths (all leaf-eaters) are often thought of as "dumb" lol
Leaf eaters also tend to have larger jaw muscles and attachments for same.
That tends to limit the size of the brain case, as that space is needed for the chewing apparatus and supporting skeletal features[0].
I remember seeing a lecture by an anthropologist who showed a graph that mapped the size of different primates vs what percent of their day they spend just chewing food. Chimps spend about 60% of their days just chewing for example. The trendline was a surprisingly effective predictor for other primates. Except for humans. Based on our mass we would expect humans to spend around 70% of our day just chewing. Instead it's less than 10%
Another point of evidence that supports the hypothesis put forth in the OP
> The researchers analyzed the brain sizes and diets of over 140 primate species spanning apes, monkeys, lemurs and lorises and found that those who munched on fruit instead of leaves had 25 percent more brain tissue, even when controlling for body size and species relatedness.
[0] https://www.npr.org/sections/thesalt/2017/03/27/521423216/wh...