Have you looked around lately? First people grow taller and then grow wider.

Also elasticity wears out. Poorly fit shirts that are worn out look pretty sloppy. And I say this as someone who wears poorly fit clothing that is worn out.