An educated guess: you get stronger negative reactions to caffeine: anxiety/panic, aggression, insomnia, heart racing and/or palpitations, sweats, tremors/muscle spasms, dehydration.

That said, I don't think it's as genetically hard-coded as the the root comment suggested.

I've been through episodes (coinciding with when I was having debilitating symptoms of chronic fatigue and depression), when I experienced these symptoms (sometimes very severely) when consuming caffeine. As I've become more physically energized and mentally well, my tolerance for caffeine has vastly improved.