It could, and you'll want to talk to labor lawyer for specifics about what is and isn't acceptable.

For example, when you talk about the candidate "seeming intelligent" that could suggest that appearances may have played a role in the evaluation, especially if you can't narrow down more specific reasons.

It's best to stick to objective factors such as the candidate's lack of experience in <some key area>.