http://www.webmd.com/allergies/features/the-weather-wreaking...
May not be the correct answer for you, but it's true for me. In the winter when the temperature has dropped suddenly I often need to take a decongestant for a few days and then I'm fine for the rest of it. Crud in your sinuses (for me at least) caused by allergies (grasses in my case) or weather can trigger a low-grade fever and other cold-like symptoms. It's worse since I work in a heated (too hot, IMO) office and then have to go back out into the cold. This was less of a problem when I worked in a more "frugal" office where the temperature (while still warmer inside) wasn't so extremely different from outside.
Do you wear a long-sleeve shirt? I used to have this problem when I wore t-shirts and worked in an air-conditioned office. Now I wear a long-sleeve button-down shirt with an undershirt and I feel much more comfortable every day.
Would love to know why this happens though?
I always figured it was somehow psychosomatic.