Our entire civilization is built on the assumption/requirement that there are enough areas of land that will remain within certain atmospheric parameters for enough contiguous months to grow a lot of food. Technology has widened the atmospheric parameters a bit, and has increased the yield enormously.
Climate change is the result of more energy in the biosphere. This energy is in the process of increasing the (effective) random variabilities of all atmospheric parameters, globally.
Our atmosphere has always been a chaotic system, and when the necessary parameters weren't met for a given region for enough time, then a lot of people migrated and starved.
Adding more energy into a chaotic system tends to make it even more chaotic.
As you noted, these permutations go in both directions.
More unusual heat waves.
More unusual cold waves.
More unusual droughts.
More unusual rain/flooding.
More unusual storms, both warm and cold.
And all of these things globally.
I think there's non-trivial chance that climate change will, on average, cut world-wide food production by a substantial amount over the next two decades. If that is indeed the case, it will be a lot of downs and ups, just as we're seeing now, but, slowly, more and more severe.
For the last few years, we've had pretty mild winters. This last one, I didn't use my radiators at all because the house is very well insulated and the temperatures didn't drop that low.
This is one of the main reasons the average 2.5 degree celsius change (which on an absolute scale would not make a huge difference in the first place, most people would hardly notice) is such a huge affair.
It can cause local variations that are a very large multiple of the 2.5 degrees, much like a heatpump does not require a whole lot of energy to move a much larger amount of energy around.