Most of my signal contracts are also unverified (my nerd friends are). However, I am notified if their key changes, so unless our connection was MITM'd on first contact, I would at least have a warning and know to be more suspicious. Usually I'll follow up with people out of band to verify that they got a new phone or something. This happens infrequently enough that it is not a hardship.

I'd prefer to verify all of my contacts but, given my security model, this is good enough and a huge step up from email.