It's definitely not clear to me that farmers' markets are worse for the environment - in many cases, it's the same farmer looking for a direct-to-consumer outlet for some of their product.
So the most optimum thing (in terms of CO2) is to plant food where it grows best and then ship it.
As an extreme example compare a greenhouse with regular outdoor planting. Or compare growing food where you need a lot of pesticides or fertilizer vs where you do not.
The reason is simply that shipping uses very very little energy, it's the rest of the production that uses a massive amount of energy.
So buying food from a local farm delivered to you via pickup truck or having a farmer's market that's just a little bit further away from you than a grocery store or buying just a little bit less at the farmers market than you would at a grocery store can negate the entire difference of having foods shipped to you from all across the world.
So, it's a truck and a freight train and a ship and a truck for foreign produced food, and only a truck for locally produced food.
Rather the farmer will grab a little of this, and a little of that. Whatever he can sell the day. This uses far more energy.
And don't forget the energy needed to harvest the food - large growers do it in bulk, small farmers will need to fire up the tractor for short runs.
A cargo ship uses FAR FAR FAR less energy than you might imagine. Even shipping something by rail across the US uses less energy than driving to the next city.
There is a middle ground between the "processed foods from these mega corporations" (which I agree you should not buy) and farmers markets though: Just buy regular produce from a grocery store, and cook normal food.