The trick to not getting into an argument with this gossiper is to always seem like you have the utmost sympathy their point of view, but you can also defend the other person by playing as if you are extremely sympathetic to them too. You will come off as just a really nice person, and most people will like you for it.
If you really want to leave the lunch, you could have "an important lunch meeting" conflict with this one (ask some other coworker to have lunch with you to ostensibly talk about work), or claim that you have an important deadline to meet and must work through lunch for several weeks. After a while, they will forget to ask you.