You're right - farmers generally didn't just let the fields lie fallow and become forested. Piece by piece, farmers sold their most marginal lands to housing developers, commercial ventures, etc.