Re:Language Models Pack Billions of Concepts into 12,000 Dim...
by aabhay on Monday September 15, 2025 @04:33AM
(#45247430)
You’re right. I gave the wrong number. My model implies 2^12000 concepts, because you choose whether or not to include each concept to form your dimensional subspace.
I’m not even referring to the values within that subspace yet, and so once you pick a concept you still get the N degrees of freedom to create a complex manifold.
The main value of the mental model is to build an intuition for how “sparse” high dimensional vectors are without resorting to a 3D sphere.
Instead, a naive mental model of a language model is to have a positive, negative or zero trit in each axis: 3^12,000 concepts, which is a much lower number than 12000!. Then in practice, almost every vector in the model has all but a few dozen identified axes zeroed because of the limitations of training time.