That is fair. Unfortunately, politics, like religion, can be deeply personal, and it is very difficult to predict how someone will react.

If the company had a proper feedback mechanism- whereby I could have left feedback for him anonymously (so that only his boss would see my name) I might have felt more comfortable bringing it up.