It's entirely relevant. You're a recent grad. You haven't been landing your interviews. And he's giving you a plausible reason why.

Your interpretation of it being insulting doesn't preclude it from being true.