First, our primate relatives can easily eat a lot of meat and prefer meat if given. They do not have a lot of meat available in wild, but can occasionally eat it in the wild too. When they have meat available, they eat brains (fat) first: https://www.nationalgeographic.com/animals/article/chimpanze...

Second, they ferment fruit because fruit is always slightly poisonous (fructose).

Third, after being close to our primate relatives for some time, our ancestors spent a lot of time hunting, which is not a trait of monkeys we know. They eat more varied meat diet than any predator known, did that for a long time, much longer than (now) we eat crops.

Fourth, we have much higher normal percentage of fat in our body. For men, the normal fat percentage range is from 18% to 22%. Our primate relatives have much less normal fat, for most primates it is less than 9%. Our primate relatives sport bodybuilder's or high performance athlete's fat range just living their life.

As with alcohol, fructose is not a toxin in small doses, because liver can handle it (especially when it is fermented into lactic acid). But it is a toxin in doses we usually experience now. Look for "non-alcoholic fat liver syndrome".

In medicine they say the dose makes the poison.

But in general it's a far more complicated causality chain. When weight gain leads to insulin resistance then sugar is poison. We have lifestyles that maximize both a lack of physical labor and an excess of high sugar foods.

And evolution proves your point as the main difference between our "close relatives" is that our gut is better at digesting richer food (aka meat), which gives us a smaller and more efficient digestive system which freed resources for our big powerful brain.