When I worked at a Jeans & T-shirt sort of place, I always told candidates that they too could dress casually. Many found the idea scary and dressed up a bit anyway.
Do you really want to hire the kind of person who can't figure out how to avoid this disadvantage?
Dressing up can be a signal that someone is desperate for work. And failure to dress up can indicate confidence and other options.
The point is it can be a positive or negative sign, depending on various things.
This may apply less for programmers and high-end technical positions, but this is the exception not the rule.
So not only don't we care how people dress at YC interviews, dressing up is actually a (minor) red flag. We'd rather you spent that effort on something else.
Some people use clothing as a means of mental preparation for entering a non-casual situation. I agree there's some empty ceremony to wearing a business suit, but I don't think that's all there is to it.
Even if attention is a limited resource, the interview usually is a place where first impressions matter, so I'd say budgeting some attention towards looks is not misguided at all.
Shouldn't we hire people who know what actually matters, and what doesn't matter, and prioritize?
You should be glad that I'm willing to signal in such a way that we find that out quickly, instead of pretending I think more like you and possibly getting hired even though I'd be a bad fit working for you.