Re:Show HN: Word2Bits – Quantized Word Vectors
by creichenbach on Tuesday March 20, 2018 @05:04AM
(#16626948)
Yeah, Levenstein distance is pretty close to our goal metric of "similarity". The thing is that we're feeding the search query into a neural network, hence we need some kind of vector represenation.
Re:Show HN: Word2Bits – Quantized Word Vectors
by igravious on Tuesday March 20, 2018 @05:43AM
(#16627087)
Need?
Or don't feed the search query directly into a neural network?
Re:Show HN: Word2Bits – Quantized Word Vectors
by creichenbach on Tuesday March 20, 2018 @08:36AM
(#16627954)
I don't understand; what do you mean?
This is one of the defining differences between Word2Vec and Fasttext. But fasttext incorporates these character vectors as part of calculating the semantic vector, so you can't expect carg and cargo to end up being similar, but people have thought of it.
I don't think partial search is that uncommon, but I don't think it is usually solved by using vector representations similarly to word vectors. It seems like what you are looking for is usually accomplished by edit-distance / Levenshtein distance [1]
[1] https://en.wikipedia.org/wiki/Levenshtein_distance