Mutual information relies on probability distributions, so I don’t think it makes sense to compare two random embeddings using it. Also, cosine similarities aren’t used in training BERT, I think it’s just cross entropy loss, which makes sense when doing classification.