Adversary of what caliber? All they have to be able to do is redirect a host (for instance by poisoning a local DNS lookup); the actual exploit is trivial. They're self-signed certificates; in the absence of a "TOFU" scheme that doesn't yet meaningfully exist, they provide no security against attackers willing to spoof DNS responses.
"TOFU" is also a truly abysmal fit for any application with a mass audience (at any given time, a huge percentage of your inbounds are going to be people making their first request, all totally open to attack --- contrast that with SSH, where you make first contact with a host on a new machine fairly rarely).
Of course, self-signed certs + TOFU theoretically allow a third party to insert itself as a MITM at the first connection. This needs a lot of tracking and preparation beforehand; no adversary of this caliber is going to be interested in Gemini content.