It's both. Herbicides wipe out soil-stabilizing fibrous root networks. Fungicides wipe out soil-stabilizing mycelial mats. Nitrogen fertilizers poison the microbial environment in healthy soil (with about a trillion organisms per cc), destroying a significant source of organic matter. Insecticides kill off the earthworms, which aerate the soil allowing it to soak rain instead of running off (taking soil with it).
Any fertilizer that's mined out of the ground is obviously transient, so the ability to have a food system without fertilizer is a Very Important Thing.
So if you want to raise cereals and vegetables, you have to choose: tillage or chemical use. The latter is much easier on the soil, even if it's not perfect.
Chemical use can be eliminated (yes, even to grow cereals!) by cultivating healthy living soil. The easiest way is with green mulches/cover crops, which serve the same thermodynamic functions but using sunlight instead of fossil fuel.
* Legumes fix nitrogen. Yep, solar powered.
* Insect predators eat pests. They're part of the food chain, so they're solar powered.
* Deep rooted plants[2] bring up minerals (essentially micro-scale mining on site in response to mineral deficiencies) and make soil. Photosynthesis, natch.
You get the idea.
I don't really have to say that an erosive food system that compromises humanity's only closed-loop habitat is a bad value, do I?
[1] http://www.rand.org/pubs/occasional_papers/OP179.html
[2] e.g. Symphytum
For example make energy cheap enough, and we can desalinate enough water to re-green the Sahara desert.
That's not something we can do today, but the future is bright, and not limited.
So all we're waiting for is "power too cheap to meter", eh? :D
Planting trees can also green deserts[1], and does so without perpetual future energy inputs from desalination. This makes sense when you realize that plants cause rain.[2] The reverse is also true - cutting down forests creates deserts.
[1] https://www.youtube.com/watch?v=reCemnJmkzI
[2] http://www.nature.com/nature/journal/v496/n7445/full/nature1...
We're only surviving today by stealing from our future. That's what "unsustainable" means. The problem is that the future catches up with you. The fact that industrial agriculture seems to work today obscures that fact.