Ad hominem is short for argumentum ad hominem, which is a type of logical fallacy. Saying "it's an ad hominem but it's not fallacious" suggests you either don't understand this, or you think that the example you've given is an example where the authority of the arguer is indeed part of the argument (which is indeed a non-fallacious ad hominem), which it's clearly not.