> Actually, the farmland will just move. Northern regions will become more arable. There will be disruption but maybe nothing we can't handle.

Even if we ignore the effects of increased variability (which makes investing in production anywhere less secure) and just consider, as you do, the averages, this is problem for people as long as countries and borders are a thing, because shifting where its possible for humans to support themselves has some pretty drastic consequences.