Re:We’re one step closer to reading an octopus’s mind
by TeMPOraL on Tuesday April 11, 2023 @06:51PM
(#35532775)
The magic is really in how absurdly high-dimensional those vectors are. The dimensionality of the latent space in current breed of LLMs is, IIRC, on the order of hundred thousand dimensions.
Now, even if all the model does is 1) turn your prompt into high-dimensional vectors, 2) run an adjacency search to find the vectors in the latent space nearest to your vector, and 3) translate those vectors back to tokens, it's more than enough for it to work with concepts. Again, we're talking 1000 to 100 000 dimensional vectors here. Any kind of semantic similarity between words you can think of (tree - green - grass, tree - tall - skyscraper, tree - data structure, tree - files, etc.) can fit in there - the relevant words (tokens) will be close together along some dimensions. So, if you pick a vector somewhere in the latent space, and look around (in hundred thousand dimensions) for its nearest neighbors, the group of points you'd be looking at is, IMHO, a concept in its raw form.
Now, even if all the model does is 1) turn your prompt into high-dimensional vectors, 2) run an adjacency search to find the vectors in the latent space nearest to your vector, and 3) translate those vectors back to tokens, it's more than enough for it to work with concepts. Again, we're talking 1000 to 100 000 dimensional vectors here. Any kind of semantic similarity between words you can think of (tree - green - grass, tree - tall - skyscraper, tree - data structure, tree - files, etc.) can fit in there - the relevant words (tokens) will be close together along some dimensions. So, if you pick a vector somewhere in the latent space, and look around (in hundred thousand dimensions) for its nearest neighbors, the group of points you'd be looking at is, IMHO, a concept in its raw form.