News for hackers. Stuff that matters.
Re:Word Embeddings: Past, Present, and Future by visarga on Monday June 12, 2017 @12:29AM (#14535203)
Great slideshow. Word embeddings are already indispensable in NLP, but I'm wondering if it is right to assign just a unique vector per word. Instead, each word should have a vector based on context, for disambiguation and nuance.

I see word meaning as an action similar to object recognition in vision - we have to infer meaning because the actual word itself is a family of meanings. Most of the time it's used one word - one embedding, even in word2vec which is the standard method used in papers.

Of course word sense disambiguation is an old topic of research and there are many methods. The translation task is probably the ultimate word-disambiguation application - it shows how much work is still to do after word embedding, in order to understand the meaning of words. If words had unique meanings, we could have done translation with simple dictionary substitutions.

Re:Word Embeddings: Past, Present, and Future by nl on Monday June 12, 2017 @01:06AM (#14535276)
Sense2Vec is a thing; Spacy has a demo https://demos.explosion.ai/sense2vec/?word=natural%20languag...

For many downstream tasks it doesn't seem to improve things though. NLP is hard - lots of things should work but don't because of the sparse training data. Like this slide deck says: most sentences occur only once.

Re:Word Embeddings: Past, Present, and Future by visarga on Monday June 12, 2017 @01:16AM (#14535309)
I realized after posting that I was naive. LSTMs are practically word embedding combinators. Training LSTMs should solve the problem of combining words and interpreting them in context.
Re:Word Embeddings: Past, Present, and Future by Jack000 on Monday June 12, 2017 @02:45AM (#14535521)
I thought Spacy only did part of speech disambiguation rather than sense disambiguation. ie. it would be confused if a word has multiple meanings, all of which are nouns.
Re:Word Embeddings: Past, Present, and Future by nl on Monday June 12, 2017 @08:22AM (#14536681)
This demo uses the Spacy POS tagging to build a word embedding which disambiguates multiple senses. The paper explains it: https://arxiv.org/abs/1511.06388
Re:Word Embeddings: Past, Present, and Future by orthoganol on Monday June 12, 2017 @03:40AM (#14535677)
> Instead, each word should have a vector based on context, for disambiguation and nuance.

You do realize even under vanilla Word2vec, each word embedding is a sum of both the word vector and a vector of a surrounding window of words? Each embedding results from maximizing the likelihood of a small surrounding context given the word. Thus, context is factored in.

Re:Word Embeddings: Past, Present, and Future by nl on Monday June 12, 2017 @08:28AM (#14536725)
This is true, but the context implicit in the vector is the average of the training data.

Vanilla Embedding based models don't handle word order well (as some of the examples upon this slide deck show).

Re:Word Embeddings: Past, Present, and Future by alexott on Monday June 12, 2017 @05:12AM (#14535956)
This article digs into some aspects of this, more related to IR: Relevance-based Word Embedding: https://arxiv.org/abs/1705.03556
Re:Word Embeddings: Past, Present, and Future by luisjgomez on Monday June 12, 2017 @08:36AM (#14536772)
Anyone have pre-print/draft version link for this cite:

Alessandro Lenci, Distributional models of word meaning, 2017

Doesn't appear to be release yet: http://www.annualreviews.org/doi/abs/10.1146/annurev-linguis...

Re:Word Embeddings: Past, Present, and Future by w4nderlust on Tuesday June 13, 2017 @05:22PM (#14548767)
I had the chance to read a preprint, it's a great survey, it will be published soon.
Re:Word Embeddings: Past, Present, and Future by natch on Monday June 12, 2017 @01:16AM (#14535311)
Re:Word Embeddings: Past, Present, and Future by paradite on Monday June 12, 2017 @02:21AM (#14535463)
Side note: You may not want to open this on mobile data. It was downloading at 1mb/sec and I had to force close the browser immediately.

Edit: Not sure how big is it in total though. Maybe someone not on mobile can share.

Re:Word Embeddings: Past, Present, and Future by w4nderlust on Tuesday June 13, 2017 @01:32AM (#14543021)
I edited the page. Now there's a download button and a preview button. If you click the preview button, the javascript document viewer embedding the pdf will be loaded, otherwise it won't be loaded.
Re:Word Embeddings: Past, Present, and Future by thomasfoster96 on Monday June 12, 2017 @02:47AM (#14535528)
On desktop Safari it was almost 10MB in total for me, 7.7MB of which is the PDF slides.

Edit: Images in the slides are downloaded as you navigate to them, they’re no more than 200KB per slide.