The good thing about being a hunter-gatherer is you don't have roots. If the local food sources are depleted you simply move on to someplace better. Agriculture roots you to the land, especially if you had to clear the land to make room for your plants and animals. Moving becomes expensive, which makes you more vulnerable to local disruptions to the environment.