There’s quite a bit of variation within those. Where I live, “casual” these days is usually something like Columbia PFG - the epitome of “dad clothes”, as my wife tells me. I tend to wear that pretty much exclusively. I substitute jeans for the shorts in winter and buy a new leather jacket about every 3-5 years. I pass my old one on to someone else.

Your point stands, though. As a man, my clothes are 100% based on practicality, and I often only discard them when they’re too worn to be presentable.