You're certainly right though, quality material and a cut that fits you well is fantastic. I have a few nice polos that I got tailored as an experiment, and they are now some of my favorite shirts. I would have more tailored, but it's $20 a pop and that's only if the shirt already fits me fairly well- which most shirts don't.
It's a question of construction decisions, not quality. For example, what fabric does it use? My denim jeans hold up to falls, rock, and dirt much better than light cotton slacks. What is the stitching pattern? Are there reinforcements? That sort of thing.
if your body changes, it's no longer going to fit.
Yup, I've been running into that problem. Thankfully the "problem" is that my biceps are stretching my sleeves, so I can feel good about said "problem".