I think the related but slightly more general claim is that you cannot update your beliefs in the same direction for both of two exhaustive and exclusive outcomes. In this case, if people disagreeing with you reinforces your beliefs, then it should be the case that everyone agreeing with you makes you question those beliefs. (This might be reasonable if you are e.g. in a mental asylum.)