I have been a lunch buddy too, sure they can told u that it was not an interview but nothing in this world is ever black and white. The person I'm having lunch is probably will be my future coworker so if this person doing something stupid or something I don't like during the lunch, there is no guarantee I will not hinted it to the hiring manager later on.