The paper you link is about word2vec, which has taken quite a while to understand. This paper shows word2vec is a particular factorization of the word-context matrix.

Neural Word Embedding as Implicit Matrix Factorization https://papers.nips.cc/paper/5477-neural-word-embedding-as-i...

Which is some of the story. I found this paper also interesting.

Towards a Better Understanding of Predict and Count Models https://arxiv.org/abs/1511.02024