Very good point. This is an active area of research, there's been some work in using other similarity measures like MI,Kendall's Tau etc in embeddings as well. And they have been found to perform pretty well.

I found something for MI if you wanna check it out : https://www.aclweb.org/anthology/2020.acl-main.741.pdf

EDIT: As @rubatuga said,they have been avoided largely as embeddings are continuous random vars and MI is for distributions. Nonetheless I think there can be merit in exploring these

PS: I am the author and just wanted to talk about the familiar basic stuff on this one