News for hackers. Stuff that matters.
Re:Understanding word vectors by visarga on Friday March 02, 2018 @11:57AM (#16503281)
Word vectors are at the same time amazing because they contain a huge amount of latent information, and not good enough because they collapse a space of very high dimensionality in ~300 dimensions, so they have a limit to how much they can discriminate between close topics. I have done a lot of experiments on classifying text in thousands of topics, and sometimes they work amazingly well, other times they are really hard to use, depending on how close together are the topics I want to discriminate between.

Another problem of word vectors is that any word might actually have multiple senses, while vectors are just point estimates. If we wanted to be correct, we needed first to find the right sense for each word in a phrase and only then assign the vector. There is research in "on-the-fly" word vectors that adapt to context, but it's much harder to use.

A third problem of word vectors is out of vocabulary words and words with low frequency. For OOV, the usual solution is to create character or character-ngram embeddings that can be used to compute embeddings for new words. For low frequency words we usually ignore them (apply a cutoff).

Then there is the problem of phrases and collocations - some words go together, such as "New York" and "give up". The meaning of the phrase is different from the sum of the meanings of the component words. In these cases we need to have lists of phrases and replace them in the original text before training the vectors, so we have proper vectors for phrases.

By the way, one amazing tool that goes with word vectors is the library 'annoy' which can do similarity search in log time. So you can do approx 1000 lookups per second per CPU even if the database contains millions of vectors, pretty good. Annoy can be used to find similar articles, or music recommendations. Another remark - my preferred word vectors are computed with Doc2VecC (a variant of doc2vec with corruption). Doc2VecC seems more apt to discriminate between topics, but the secret is to feed it gigabytes of text.

Playing with word vectors has taught me intuitively how it is to navigate a space of high dimensionality. It feels different than 3d-space because each point has a shortcut to other points, each point leads to hundreds of other places which might be far apart. It's like a kaleidoscope where a small change can create a very different perspective.

doc2vecC: https://github.com/mchen24/iclr2017

annoy: https://github.com/spotify/annoy

Re:Understanding word vectors by aisofteng on Saturday March 03, 2018 @01:07PM (#16510587)
>Another problem of word vectors is that any word might actually have multiple senses, while vectors are just point estimates. If we wanted to be correct, we needed first to find the right sense for each word in a phrase and only then assign the vector. There is research in "on-the-fly" word vectors that adapt to context, but it's much harder to use.

There has been work on representing words not as vectors but as multimodal Gaussian distributions in order to try to deal with polysemy, such as [0], of which an implementation (which I have not tried to use) is available on GitHub at [1].

>It feels different than 3d-space because each point has a shortcut to other points, each point leads to hundreds of other places which might be far apart. It's like a kaleidoscope where a small change can create a very different perspective.

I appreciate that the author of the parent comment has found some sort of intuition, but I would caution others from trying to use the above quote in order to develop their own intuition as it is meaningless in any rigorous sense.

[0] https://arxiv.org/abs/1704.08424

[1] https://github.com/benathi/word2gm

Re:Understanding word vectors by aglionby on Friday March 02, 2018 @03:06PM (#16505188)
> Another problem of word vectors is that any word might actually have multiple senses, while vectors are just point estimates. If we wanted to be correct, we needed first to find the right sense for each word in a phrase and only then assign the vector. There is research in "on-the-fly" word vectors that adapt to context, but it's much harder to use.

This is interesting, and there seems to be a bit of debate about it (at least with compositional distributional semantic models). [0][1] seem to show that sense disambiguation helps in some contexts, and [2] show that they don't in others. It doesn't seem immediately clear who is right here. I agree with you though that it seems pretty likely that disambiguating would be helpful.

[0] https://www.aclweb.org/anthology/W13-3513

[1] https://www.aclweb.org/anthology/P16-1018

[2] https://www.aclweb.org/anthology/D10-1115

Re:Understanding word vectors by kevinwang on Friday March 02, 2018 @03:16PM (#16505286)
It seems obvious to me that disambiguating _correctly_ would necessarily improve performance. As an extreme example, homonyms like bark (the verb) and bark (on a tree) have nothing to do with each other, and ideally would be considered two different words.
Re:Understanding word vectors by taeric on Friday March 02, 2018 @07:01PM (#16506868)
I imagine contranyms would be among the most difficult here. Would be fun to see how these vectors stack up with those.
Re:Understanding word vectors by kevinwang on Friday March 02, 2018 @08:19PM (#16507236)
Well, antonyms are usually right next to each other in word embedding spaces anyways. (since the context they're used in is often hard to tell apart)
Re:Understanding word vectors by taeric on Saturday March 03, 2018 @11:40AM (#16510061)
Contranyms would be identical in the embedding space, no?

I'm actually intrigued that nobody has made a text processor named after https://en.wikipedia.org/wiki/Amelia_Bedelia yet. :)

Re:Understanding word vectors by aglionby on Friday March 02, 2018 @03:21PM (#16505326)
Yeah I would agree. I've struggled to find much work that looks at this at a large scale though!
Re:Understanding word vectors by nicklovescode on Friday March 02, 2018 @06:32PM (#16506721)
AllenNLP’s recent work ELMo is very good at this task. Check it out!
Re:Understanding word vectors by aglionby on Friday March 02, 2018 @06:45PM (#16506783)
Much appreciated! Will have a look.
Re:Understanding word vectors by nl on Friday March 02, 2018 @08:05PM (#16507163)
Actually...

This seems like it should obviously make sense, but in practice it doesn’t necessarily. Usually the sense in defined by the context, and so the vector can carry the meaning of multiple contexts at once.

There are some specific tasks where this breaks for some words. I can’t remember one I’ve seen right now, but thinking about the “bark” example, it can cause a problem when both the “woodyness” and “noisyness” of a word cause completely different results. In practice that’s pretty rare.

Generally though, the non-relevant meanings don’t have any negative effects as they can be ignored.

Re:Understanding word vectors by RoboTeddy on Friday March 02, 2018 @02:43PM (#16504933)
> we needed first to find the right sense for each word in a phrase and only then assign the vector

Interesting! I wonder if you could you e.g. arbitrarily split a word into some number of symbols, e.g. two, and each time you're going to apply a training update, only apply the update to one of the symbols -- perhaps initially choosing the symbol facing the greatest loss (forcing the symbols apart in vector space), and then eventually switching over to picking the symbol with the smallest loss (letting each settle onto its own precise meaning)?

Re:Understanding word vectors by hiker512 on Friday March 02, 2018 @12:23PM (#16503595)
Can you recommend an implementation or paper for handling OOV words via character level embeddings?
Re:Understanding word vectors by patelajay285 on Friday March 02, 2018 @12:28PM (#16503644)
We just open sourced a easy to use library called Magnitude that handles out-of-vocabulary words and uses Annoy indexing for fast most_similar queries for word2vec, GloVE, and fastText:

https://github.com/plasticityai/magnitude

Re:Understanding word vectors by Sinidir on Tuesday March 06, 2018 @10:53AM (#16529549)
Thank you very much. This might help with my Master Thesis :)
Re:Understanding word vectors by visarga on Friday March 02, 2018 @12:35PM (#16503735)
Great work!
Re:Understanding word vectors by fnl on Saturday March 03, 2018 @09:28AM (#16509374)
I agree - and also to your initial comments on the actual benefits & limitations of word vectors, that I can 100% subscribe to.
Re:Understanding word vectors by wenc on Friday March 02, 2018 @02:14PM (#16504720)
Just an FYI, FastText's default implementation handles OOV words via word n-grams and character n-grams. (see switches -minn, -maxn, and -wordNgram)

https://fasttext.cc/docs/en/options.html