This is exactly the argument for privacy. Privacy is about agreed lines because we know that there are some places where we just won't get along. The "if you have nothing to hide" argument assumes that you want to see the things that I'm hiding. In this case, people are hiding their political affiliation (or at least their explicitly expressed opinions) because we know that if it was shared, all of our lives would be harder.