You can also wear a fleece everywhere. It's light and comfortable. But it just doesn't look as good and people will judge you differently for it in some contexts.

Clothing is just as much about comfort as it is about impressions.