Re:Show HN: I mapped HN's favorite books with GPT-4o
by nostrebored on Tuesday September 10, 2024 @01:24AM
(#41497413)
How behaved they are visually depends on what drives variance and what you’re hoping to see. There are certainly some nice properties in some dimensionality reductions, but if you flatten a space of faces it’s less likely that you’ll get the property of “brown hair” as a query embedded in any visually interesting way than actually putting in a face as a query.
More clearly, symmetric retrieval is easier to visualize in a dimensionality reduced space than asymmetric retrieval.
I suspect that some form of multi vector document embedding would be more understandable in the reduced space than this single vector representation.
IMHO it's a category error that results from tutorials using the king + female = queen example (which, funnily enough, wasn't even true for the original word2vec, if commentary I've read previously here is correct).
Working with them a lot has me picture them more as "a multivariate function that outputs 768 numbers, and was learned by brute force" than "something that sees in 768 dimensions" --- of course, they're both true, but the second interpretation shades more than it illuminates once you're past the very first interrogatory of "so what is this calculating, exactly?"