In medicine they say the dose makes the poison.

But in general it's a far more complicated causality chain. When weight gain leads to insulin resistance then sugar is poison. We have lifestyles that maximize both a lack of physical labor and an excess of high sugar foods.