When they come round and ask for a chat you can easily decline to let them in. That's just a personal choice. Personally I would prefer to keep them out becuase of my belief (which may be wrong) about warrants to search.

The reason you talk to them is to get information. "We've had this complaint. We're letting you know." In this case I genuinely have no idea why the police felt it was appropriate to get involved and I would send a complaint in, even though they had been very polite. In other cases they may tell you that some person feels harassed by you. Ignoring what the police say could lead to you being arrested for them to question you about the harrassment; or maybe the other person will get a non-molestation order; or etc.

Tl;dr in England we're policed with our consent and most people kind of feel that policing should be collaborative. Most people don't mind polite chats; it's petty little officious police being obnoxious during a traffic stop that is annoying.