Re:Facebook releases 300-dimensional pretrained Fasttext vec...
by webmaven on Saturday March 04, 2017 @08:10PM
(#13794003)
> it is still somewhat surprising to me that you can get as good results as people claim without a nearly infinitely dimensional space
As I understand it, the maximum number of dimensions required is equal to the number of words. That is, if you did no dimensional reduction, you have a vector that expresses exactly how close occurrences of the word in question are on average to occurrences of each and every other word.
That's a very large number of dimensions, but hardly infinite.
Reducing the number of dimensions turns "distance from every other word" into "distance from abstract concepts", except that "abstract concept" is overstating the case, as the "concepts" aren't features of human cognition per-se except to the extent that those features are reflected statistically in the corpus that was used. Besides, the choice of the number of dimensions to reduce to is somewhat arbitrary, and no one knows right now what the "correct" number is, or even if there is a "correct" number. I'm not even sure whether much work has been done on the sensitivity of the models to the number of dimensions.
There is probably a lot of productive work to be done on dimensionality reduction techniques that make the reduced dimensions map better to abstractions that a human would recognize, at least faintly, as well as work to create corpora that better sample the full range of human expression in as compact a size as possible.
As I understand it, the maximum number of dimensions required is equal to the number of words. That is, if you did no dimensional reduction, you have a vector that expresses exactly how close occurrences of the word in question are on average to occurrences of each and every other word.
That's a very large number of dimensions, but hardly infinite.
Reducing the number of dimensions turns "distance from every other word" into "distance from abstract concepts", except that "abstract concept" is overstating the case, as the "concepts" aren't features of human cognition per-se except to the extent that those features are reflected statistically in the corpus that was used. Besides, the choice of the number of dimensions to reduce to is somewhat arbitrary, and no one knows right now what the "correct" number is, or even if there is a "correct" number. I'm not even sure whether much work has been done on the sensitivity of the models to the number of dimensions.
There is probably a lot of productive work to be done on dimensionality reduction techniques that make the reduced dimensions map better to abstractions that a human would recognize, at least faintly, as well as work to create corpora that better sample the full range of human expression in as compact a size as possible.