At my last gig at Stripe we had a candidate fail the lunch interview because he spent much of the time bad mouthing his current co-workers. Not a good look. Unless there is a red flag though, the lunch interviewer doesn't have to show up for the post-interview discussions.