It does sort of involve crypto: if the page you control weren't served over HTTPS it wouldn't be too hard (DNS poisoning) for someone else to trick a server into verifying the wrong user.
If DNS poisoning is so easy why don't you perform the same attach on Let's Encrypt since that also uses plain HTTP.
Edit: supported since 2018 https://github.com/mastodon/mastodon/pull/8703