There is still a risk that someone is running a MITM attack. The initial conversation would be authentic, but the key belongs to someone else who is just forwarding the messages. Your communications would no longer be private and they could switch from passive eavesdropping to impersonation at any point without changing the key.