Imagine that we made word vectors out of PCA reduced sparse tf-idf or countvectorized vectors. I can tell you exactly what each PCA component explains. I could even do that at the word level because it's not difficult to do inverse transforms with some simple dimensionality reduction techniques

The model interpretability goes out the window because we used techniques for the vectorization that kinda suck. NLP is obsessed with self-supervision unnecessarily when they should be innovating in dimensionality reduction techniques