Re:Why do we use word embeddings in NLP?
by Der_Einzige on Tuesday September 10, 2019 @12:26PM
(#20929956)
Imagine that we made word vectors out of PCA reduced sparse tf-idf or countvectorized vectors. I can tell you exactly what each PCA component explains. I could even do that at the word level because it's not difficult to do inverse transforms with some simple dimensionality reduction techniques
The model interpretability goes out the window because we used techniques for the vectorization that kinda suck. NLP is obsessed with self-supervision unnecessarily when they should be innovating in dimensionality reduction techniques
Re:Why do we use word embeddings in NLP?
by edmundsauto on Tuesday September 10, 2019 @08:44PM
(#20935476)
Why do you think NLP practictioners are focusing on self-supervision instead of dimensionality reduction?
Re:Why do we use word embeddings in NLP?
by DoctorOetker on Wednesday September 11, 2019 @02:05PM
(#20942975)
I agree, and I have an idea for this dimensionality reduction which makes the original unsupervized word vectors interpretable.
it boggles my mind I haven't seen anyone implement my idea.
Re:Why do we use word embeddings in NLP?
by aglionby on Thursday September 12, 2019 @08:03AM
(#20950458)
SVD has been used for dimensionality reduction of co-occurrence matrices for ages [1], but the resulting word embeddings aren't as performant as those of word2vec/etc. The same is probably true of using PCA.
Word2vec's popularity is the result of people valuing performance (i.e. accuracy) more than interpretability.
Re:Why do we use word embeddings in NLP?
by gattilorenz on Wednesday September 11, 2019 @05:36PM
(#20945002)
Well, it might be because it's hard to read your mind from here.
Re:Why do we use word embeddings in NLP?
by DoctorOetker on Thursday September 12, 2019 @04:11AM
(#20949155)
no, that wouldn't be mindboggling
what's mindboggling to me is that I haven't seen anyone else come up with the idea independently.
Re:Why do we use word embeddings in NLP?
by DoctorOetker on Wednesday September 11, 2019 @07:02PM
(#20945733)
it's so obvious one wouldn't have to read my mind, it's all implicit in the king-man+woman=queen type of relations... if you really ask a second time, fuck it, im not in ML sector, perhaps I just give away the idea...
I'd give a bit more of a nuanced view here -- we can choose any number of properties (dimensions) to represent words, which are all learned from a corpus. 300 dimensions is a pretty popular choice. These dimensions aren't (generally) interpretable: they represent latent properties. In other words, it's not possible to say which property each dimension represents, it's simply one that your word embedding algorithm has picked up in the data. Generally speaking, feature importance is hard to define for the same reason.