News for hackers. Stuff that matters.
Re:Facebook releases 300-dimensional pretrained Fasttext vec... by saurik on Saturday March 04, 2017 @01:41PM (#13792216)
Sometimes, context can be ripe for the picking while simultaneously leaving your subject rotting on the vine.

Regardless: I think you answered a question more about "what" than "why"; I know what these models are, and how they are used, but it is still somewhat surprising to me that you can get as good results as people claim without a nearly infinitely dimensional space (and honestly I kind of wonder if this might be more a model of "the kinds of questions humans most often like to ask about words when confronted with a word for the first time or wanting to test a data set" instead of "a true understanding of what is being said encoded as vectors", allowing things like "have opposite gender" to be extremely functional vectors but probably leaving much more important concepts like "are classic opponent in war" on the floor, which is really important semantic information that isn't necessarily transtive and might not even be commutative, a thought process that seems to align with complaints about word2vec from ConceptNet.)

Put differently: I bet the set of 300 axes is actually a more useful result (though one that is more opaque and I don't hear much about attempts to analyze; but I am currently not in this field and haven't been paying attention to the literature) than the actual vector mapping (which is what people always seem excited about). I would love to see more talk of "what questions are these models weirdly good at answering versus questions where they seem so limited as to almost be useless".

Re:Facebook releases 300-dimensional pretrained Fasttext vec... by webmaven on Saturday March 04, 2017 @08:10PM (#13794003)
> it is still somewhat surprising to me that you can get as good results as people claim without a nearly infinitely dimensional space

As I understand it, the maximum number of dimensions required is equal to the number of words. That is, if you did no dimensional reduction, you have a vector that expresses exactly how close occurrences of the word in question are on average to occurrences of each and every other word.

That's a very large number of dimensions, but hardly infinite.

Reducing the number of dimensions turns "distance from every other word" into "distance from abstract concepts", except that "abstract concept" is overstating the case, as the "concepts" aren't features of human cognition per-se except to the extent that those features are reflected statistically in the corpus that was used. Besides, the choice of the number of dimensions to reduce to is somewhat arbitrary, and no one knows right now what the "correct" number is, or even if there is a "correct" number. I'm not even sure whether much work has been done on the sensitivity of the models to the number of dimensions.

There is probably a lot of productive work to be done on dimensionality reduction techniques that make the reduced dimensions map better to abstractions that a human would recognize, at least faintly, as well as work to create corpora that better sample the full range of human expression in as compact a size as possible.