On a different note, at my company, talking about work at lunch is pretty taboo unless the topic of the lunch was previously disclosed by the initiator to be work-related. Lunch is when we talk about all kinds of other bullshit and only talk about work if we're desperately spinning our wheels on something or want to vent.
The company-supplied lunches lead to useful conversations in a place where you are able to judge their utility.
When people from your office go out and do not invite you, you are unable to know who is talking to whom, or what they are talking about. And you should not know, because that would make you a spying creep.
You can't discount opportunity costs. If Oprah was giving away cars to everyone who went out to eat lunch at a local strip mall, you can't reasonably say that a person who found a quarter on the break room floor benefited by staying in the office to eat lunch.
You only observed the conversation that you observed. You did not observe all the potential conversations that might have occurred instead of the one you heard. The value of such conversations is the result of an expected value calculation, based on the potential benefit multiplied by the possibility that such benefit would be realized. That involves a lot of guesswork, obviously.