News for hackers. Stuff that matters.
Re:Understanding word vectors by aglionby on Friday March 02, 2018 @03:06PM (#16505188)
> Another problem of word vectors is that any word might actually have multiple senses, while vectors are just point estimates. If we wanted to be correct, we needed first to find the right sense for each word in a phrase and only then assign the vector. There is research in "on-the-fly" word vectors that adapt to context, but it's much harder to use.

This is interesting, and there seems to be a bit of debate about it (at least with compositional distributional semantic models). [0][1] seem to show that sense disambiguation helps in some contexts, and [2] show that they don't in others. It doesn't seem immediately clear who is right here. I agree with you though that it seems pretty likely that disambiguating would be helpful.

[0] https://www.aclweb.org/anthology/W13-3513

[1] https://www.aclweb.org/anthology/P16-1018

[2] https://www.aclweb.org/anthology/D10-1115

Re:Understanding word vectors by kevinwang on Friday March 02, 2018 @03:16PM (#16505286)
It seems obvious to me that disambiguating _correctly_ would necessarily improve performance. As an extreme example, homonyms like bark (the verb) and bark (on a tree) have nothing to do with each other, and ideally would be considered two different words.
Re:Understanding word vectors by taeric on Friday March 02, 2018 @07:01PM (#16506868)
I imagine contranyms would be among the most difficult here. Would be fun to see how these vectors stack up with those.
Re:Understanding word vectors by kevinwang on Friday March 02, 2018 @08:19PM (#16507236)
Well, antonyms are usually right next to each other in word embedding spaces anyways. (since the context they're used in is often hard to tell apart)
Re:Understanding word vectors by taeric on Saturday March 03, 2018 @11:40AM (#16510061)
Contranyms would be identical in the embedding space, no?

I'm actually intrigued that nobody has made a text processor named after https://en.wikipedia.org/wiki/Amelia_Bedelia yet. :)

Re:Understanding word vectors by aglionby on Friday March 02, 2018 @03:21PM (#16505326)
Yeah I would agree. I've struggled to find much work that looks at this at a large scale though!
Re:Understanding word vectors by nicklovescode on Friday March 02, 2018 @06:32PM (#16506721)
AllenNLP’s recent work ELMo is very good at this task. Check it out!
Re:Understanding word vectors by aglionby on Friday March 02, 2018 @06:45PM (#16506783)
Much appreciated! Will have a look.
Re:Understanding word vectors by nl on Friday March 02, 2018 @08:05PM (#16507163)
Actually...

This seems like it should obviously make sense, but in practice it doesn’t necessarily. Usually the sense in defined by the context, and so the vector can carry the meaning of multiple contexts at once.

There are some specific tasks where this breaks for some words. I can’t remember one I’ve seen right now, but thinking about the “bark” example, it can cause a problem when both the “woodyness” and “noisyness” of a word cause completely different results. In practice that’s pretty rare.

Generally though, the non-relevant meanings don’t have any negative effects as they can be ignored.