Well the problem is the damage goes exponential. Roads degrade with pressure changes and friction. But cracks let water in, which can result in erosion (of the road or underneath that supports the road), and then any elevation changes can greatly increase wear (for instance a 50,000 pound truck going 50 mph is regularly hitting the far side of the pothole).

Even a single winter can make a dramatic change to average road quality, thus the hurry every summer to repair roads.