Re:Llama.cpp 30B runs with only 6GB of RAM now
by thomastjeffery on Sunday April 02, 2023 @05:26PM
(#35415417)
I'm really just speculating here.
Because the text we write is not evenly distributed random noise, what we encode into it (by writing) is entropy.
Because LLMs model text with inference, they model all of the entropy that is present.
That would mean that the resulting size would be a measure of entropy (sum of patterns) divided by repetition (recurring patterns). In this count, I would consider each unique token alone an instance of the identity pattern.
Because the text we write is not evenly distributed random noise, what we encode into it (by writing) is entropy.
Because LLMs model text with inference, they model all of the entropy that is present.
That would mean that the resulting size would be a measure of entropy (sum of patterns) divided by repetition (recurring patterns). In this count, I would consider each unique token alone an instance of the identity pattern.
So to answer both questions: yes.