It may be that it takes less effort to get you to sufficient fitness at that age, but it's nowhere near your peak. Unless you follow an elite athlete training program, it'll take you well over a decade of exercise to max out your potential. E.g. for muscle strength, most peoples bodies simply isn't capable of synthesising muscle fast enough in even a decade to max out your potential unless you use performance enhancing drugs.

You may have a point about average fitness and rate of improvement, though.