I had the same realisation years ago when I was going through a bit of a James Bond phase. I realised that all of Bond's clothes that aren't suits are block colours, with no embellishments.

As for unbranded workout clothes, you can always try a military surplus store. Not all of the clothes there are camo style.