It's possible larger varieties previously existed and fell dormant, then later the domestic varieties started with a smaller fruit and the genetics for larger fruit were no longer in the active gene pool of known varieties.