"I heard somebody say he harassed women, so he must have done so" is also not as strong an attack as you seem to think.