It comes back at us in all kinds of weird ways, too. Biofuel production, for example. Farmers make choices all the time to switch their production to be maximally profitable (example: Almost all of NZ shifting from lamb to milk production in the past few decades).

Biofuel production puts a spanner in things because previously we were swapping some form of consumable calories for another form of consumable calorie. But now you've got fuel production competing with food production, and their prices will begin to compete similarly. That inevitably leads to either cheaper fuel (spoiler: still not great for emissions) or significantly more expensive staple foods.

So any kind of squeeze on fossil fuel pricing leads to a down-stream increase on food pricing, one greater than just the energy use for production/transportation.