If it's defined as the people you hate, then size matters a huge amount. If you only hate and would discriminate against two people, then you're amazingly tolerant.

The paradox of tolerance is not particularly difficult to deal with. You prioritize the preservation of as much tolerance as possible. If someone is trying to reduce the levels of tolerance in the world by a large amount, it's important to fight them, even if it means minor short term intolerance. And even then you tolerate them in all other ways.