Adversary of what caliber? All they have to be able to do is redirect a host (for instance by poisoning a local DNS lookup); the actual exploit is trivial. They're self-signed certificates; in the absence of a "TOFU" scheme that doesn't yet meaningfully exist, they provide no security against attackers willing to spoof DNS responses.

"TOFU" is also a truly abysmal fit for any application with a mass audience (at any given time, a huge percentage of your inbounds are going to be people making their first request, all totally open to attack --- contrast that with SSH, where you make first contact with a host on a new machine fairly rarely).