Differences between the word2vec paper and its implementation

Re:Differences between the word2vec paper and its implementa... by utopcell on Monday June 03, 2019 @07:19PM (#20089996)

I think it very unfair to the original set of word2vec papers to be talking about 'academic dishonesty'. This is a case of a user that has little to no experience with neural networks. There are a ton of articles describing the need for random initialization [1][2]. In fact, if one spends a few seconds thinking about it, the need is evident. Without it, the NN cannot perform symmetry breaking: If inputs are set to zero, all neurons will perform the same calculations, rendering the network useless.

[1] google: "neural networks vector initialization" [2] http://deeplearning.ai/ai-notes/initialization/

[ Parent | on HN ]

Re:Differences between the word2vec paper and its implementa... by bollu on Monday June 03, 2019 @07:22PM (#20090029)

That's hardly the point of the article --- the actual paper does not describe the use of two separate vectors for each word. The initialization was an interesting tidbit.

[ Parent | on HN ]

Re:Differences between the word2vec paper and its implementa... by exgrv on Monday June 03, 2019 @07:28PM (#20090080)

Except it does? After Equation 2: "v_w and v'_w are the input and output vector representations of w."

[ Parent | on HN ]

Re:Differences between the word2vec paper and its implementa... by andreasvc on Tuesday June 04, 2019 @11:27AM (#20095895)

bollu, multiple people have shown that your claim that the paper doesn't match the code is flat out wrong. I think at this point you should issue a retraction of your wildly inappropriate suggestion of academic dishonesty.

[ Parent | on HN ]

Re:Differences between the word2vec paper and its implementa... by utopcell on Monday June 03, 2019 @11:10PM (#20091438)

Of course it does. For word w they are v_w and v'_w on Eq.4 for the case of SGNS [1].

[1] https://papers.nips.cc/paper/5021-distributed-representation...

[ Parent | on HN ]