Yes, if they add a wiretapping key to one or both of your key lists, which is silent/invisible to the sender.