I have been a lunch buddy too, sure they can told u that it was not an interview but nothing in this world is ever black and white. The person I'm having lunch is probably will be my future coworker so if this person doing something stupid or something I don't like during the lunch, there is no guarantee I will not hinted it to the hiring manager later on.
The lunch buddy doesn't get the option to enter feedback for a candidate and often doesn't even get invited to the interview debrief. They really are there just for lunch.
Source: have been a lunch buddy several times.