News for hackers. Stuff that matters.
Re:LLMs use a surprisingly simple mechanism to retrieve some... by Acumen321 on Thursday March 28, 2024 @12:49PM (#39854169)
Quantization in this context is the precision of each value in the vector or matrix/tensor.

If the model in question has a token embedding length of 1024, even if it was a 1 bit quantization, each token has 2^1024 possible values.

If the context length is 32,000 tokens, there are 32,000^2^1024 possible inputs.

There are no comments on this comment yet.