In an ideal world where we have infinite time to consider all comments, you are right.

However if one has to choose what to spend time on, it is prudent to filter, and for blatant subjectivity to be used as one such filter, as it is reasonable to consider it likely that arguments made by someone who is so obviously biased against someone in the first place is more likely to be careless about the arguments he makes against their ideas.

So a a time saving exercise, it is absolutely relevant to consider other factors than the argument itself.

And yes, it is imperfect and will result in a lot of people discounting valid concerns.