1. You are assuming "better farms" have the capital and will available to purchase the land.
2. Just because a competent farmer purchases the land does not make the land arable. It could take decades to undo the damage from improperly cared for land.
I don't think we can really know anything for sure about point 2 for hypothetical internet discussion agricultural ground. It is probably true that some land has been so damaged, but it is also possible for farmer B to have better results than farmer A on the same land. This is possible even if they are in some sense equally skilled and capitalized. It doesn't rain the same every year...
Of course there is always information asymmetry, but farm buyers are aware of it. In most cases the buyer already lives in the area, so they know just be driving by over the years what really happens.
Small towns happen all over because of farming reasons, but even then access to trade was a consideration, but only secondary to close to farms. (which is to say if there was good farm land far from any way to trade there will still be a town someplace - you see this more with mines as mineral often are in places that are difficult to get to by trade, while farm countries implies enough water which implies rivers)
Farms fail all the time for a variety of reasons, but even successful farms can be less valuable than suburban residential subdivisions. If the farms in question were further from the city, better farmers would buy/rent them and the land would be productive again.