Average increase of temperature has negative consequences, but the accompanying variance increase has a lot more catastrophic impact. Storms, forest fires, floods, droughts are some of those things where increased variance can screw societies over.

A lot of this can push insurance costs so high, that insurance companies stop working. There are only so many times insurance can payout damage due to extreme storms, or crop failure. Once insurance stops, there is rapid deterioration of infrastructure.