When a debater makes a lot of bad points, extrapolating to be suspicious of any point he makes is not and ad-hominem.

And ad hominem is when you dismiss his points because he might have done something immoral in a completely unrelated field.