You either transport the food or you transport the fertilizer, heavy equipment, water, and other inputs necessary to grow the food. And often to places that aren't well suited to it.

Something's gonna have to be transported one way or another. Or we can just decide everyone will need to have blander and less varied diets. That will bring the amount of stuff to transport down, but you're still probably going to be more efficient having the inputs in a few places that can grow food really well and exporting it to the places that can't.