I've done that too, in the same situation. But I think it may backfire. Some investors may be nostalgic for their days in a startup. Others are still looking for that "startup edge" which differentiates from stagnant corporations.
If your clothing says "corporate", you may send the wrong message and drain some interest.
Obviously this varies by area and industry.
When I was in a small, casual start-up, they asked us to dress nicely for just one day to impress some visiting investors. None of us had a problem with it; we all had a stake in the company's continued health. It seemed like the natural thing to do, and I'd do it again if needed.