The difference is intellectual honesty. If you're truly willing to consider the evidence, and not just seek out things to reinforce your pre-existing conclusion, then you can be skeptical. Otherwise you're just contrarian.

The problem is that if somebody is dishonest, there's no way to convince them that they're being dishonest. Either they know it, and refuse to admit it, or they're so dishonest that they've completely lost track of what it means to objectively evaluate evidence.

So it's the kind of thing that experts in the domain will all be able to discern, while those same experts will be thought of as conspirators by the contrarian (and reinforced by their equally non-expert circle of trusted acquaintances). About the only hope you have at that point is noticing when the conspiracy is so large that it's simply not reasonable to expect it to persist. But since they get reinforcement from their large circle of non-experts, they usually conclude that the same logic applies both ways.