I agree, but I suspect that you’d have much better luck if you wore something that was superficially similar to the kinds of things other people wore, but was much better fitted and higher quality. For instance, if you showed up in a nice pair of chinos and a tailored buttoned shirt (of appropriate formality), that might come across as being really put together rather than ignoring subtle social cues by dressing in something that stands out by not fitting in.