News for hackers. Stuff that matters.
Re:Word vectors are awesome but you don’t need a neural netw... by make3 on Wednesday October 18, 2017 @07:21PM (#15504145)
professionals don't use pretrained word2vec vectors in the really complex (like neural machine transnation) deep learning models anymore, they let the models train their own word embeddings directly, or let the models learn character level embeddings.
Re:Word vectors are awesome but you don’t need a neural netw... by dontreact on Wednesday October 18, 2017 @09:48PM (#15504856)
This is false. Sometimes you have limited data for your task and initializing from pretrained embeddings boosts performance.
Re:Word vectors are awesome but you don’t need a neural netw... by arrmn on Wednesday October 18, 2017 @07:26PM (#15504178)
What exactly do you mean with "they let the models train their own word embeddings", can you elaborate more on this or are there any current papers about this topic?
Re:Word vectors are awesome but you don’t need a neural netw... by make3 on Wednesday October 18, 2017 @07:33PM (#15504220)
the embedding layer is the layer that converts the one hot word feature in to a continuous multi dimensional vector that the deep net can learn with. they used to pretrain that layer separately with word2vec. now as it's just a neural net layer, they let the translation model train it with backprop on the main (translation /dialog / qa, etc) task as a regular layer
Re:Word vectors are awesome but you don’t need a neural netw... by oh-kumudo on Thursday October 19, 2017 @02:10PM (#15510170)
I would say it depends on your task, if pre-trained ones performs better, you use pre-trained ones, otherwise you let the model learns the embedding itself.