News for hackers. Stuff that matters.
Re:Word embeddings in 2017: Trends and future directions by jph00 on Saturday October 21, 2017 @07:59PM (#15524303)
Word embeddings (or subword embeddings) are used for nearly all recent NLP algorithms, both shallow (e.g. FastText) and deep (e.g. Google Neural Translation). Unless you're using a basic bag-of-words approach, you need to translate your words into some vector format, so you probably want some kind of embeddings. In practice, all the state of the art approaches for translation, language modeling, classification (eg sentiment analysis), etc. all sit on top of embeddings.

It's not word embeddings job to handle phrases - but nearly all modern phrase embedding algorithms sit on top of word embeddings. They often create a weighted average of embeddings by using an attention model, or they can use a more complex model such as an LSTM with attention (e.g. CoVE - https://arxiv.org/abs/1708.00107 ).

Word embeddings can handle polysemy - high dimensional vectors can (and do) hold information of various types that is used in different contexts in different ways. Some approaches deal with this more directly (e.g. including part-of-speech as part of the vocab item), and that sometimes can help a bit.

Re:Word embeddings in 2017: Trends and future directions by PaulHoule on Saturday October 21, 2017 @08:22PM (#15524401)
Maybe I'm not reading it right, but that arXiv paper about CoVE doesn't seem to be getting anywhere near commercially useful results.

For instance, the random result for ImDB in Table 2 is 88.4 and the best one is 92.1; that's really not a lot of lift. I could see TREC-6 and TREC-50 results being good enough to let off the leash, but I still have a hard time picturing this being useful in the real world.

Re:Word embeddings in 2017: Trends and future directions by jph00 on Sunday October 22, 2017 @11:53AM (#15527317)
Oh BTW the "random" result means randomly initialized vectors. It's still using embeddings, just not pretraining
Re:Word embeddings in 2017: Trends and future directions by jph00 on Sunday October 22, 2017 @11:50AM (#15527307)
I have a version that gets 94.5 on IMDB although not published yet.