That would have to be someone actively engaging in MitM and not merely listening and keeping record.

Even if that's the case though, you can raise the difficulty by doing it through a phone call.