I'm considering the possibility it's just a humorous and slightly absurdist excuse to not remove the hat. Or an inside joke. The real reason might be more convoluted and/or personal, like having a gorbachevesque birthmark the person is not entirely comfortable with or whatever.