One issue too is with the embeddings, word vector embeddings lead to issues with polysemy, and BERT embeddings in the last layer tend to all point in about the same direction for each batch.