It depends entirely on what sort of clothes they are. Wool will insulate you even when soaked, but wearing a pair of wet jeans is worse than wearing nothing at all.

They don't provide insulation, and only serve to make you wet and extract heat from you. Just like a wet towel wrapped around a champaign bottle.