Re:Large Language Models: Scaling Laws and Emergent Properties
by SideburnsOfDoom on Saturday April 22, 2023 @03:58AM
(#35664335)
Here are some bounds, from most theoretical to most practical:
The size of the earth's biosphere
The size of the largest animal.
The size of a human skull cavity.
The size of a human skull cavity, divided by the number of neurons therein ( a few Billion).
Is this not obvious? There's nothing futile about stating these obvious facts.
Do not fool around with "if I move a potassium ion farther away, isn't that a different bound" ? correct, but moving the bound slightly, is irrelevant to the obvious statement that there are still bounds, and they are practical.
Stating that the neuron is not well-defined is similar irrelevance, if you want to precisely measure the bound, you will need to know precisely what is bounded, but I did not, merely stated the obvious truth that every such definition will come with a bound. One can say "definitely smaller than this" in a wide range of ways, without defining the exact edge.
What's the alternate argument: "I won't define it, therefor it might be larger than the solar system". This is not a serious thing to say.
And to think that such bounded systems "have infinite states would thereby be capable of storing infinite information" is pseudo-mystical foolery. Large, or even "vast", is not infinite. And as sibling comment has mentioned, there is a "noise floor" below which analog resolution means nothing.
The size of the earth's biosphere
The size of the largest animal.
The size of a human skull cavity.
The size of a human skull cavity, divided by the number of neurons therein ( a few Billion).
Is this not obvious? There's nothing futile about stating these obvious facts.
Do not fool around with "if I move a potassium ion farther away, isn't that a different bound" ? correct, but moving the bound slightly, is irrelevant to the obvious statement that there are still bounds, and they are practical.
Stating that the neuron is not well-defined is similar irrelevance, if you want to precisely measure the bound, you will need to know precisely what is bounded, but I did not, merely stated the obvious truth that every such definition will come with a bound. One can say "definitely smaller than this" in a wide range of ways, without defining the exact edge.
What's the alternate argument: "I won't define it, therefor it might be larger than the solar system". This is not a serious thing to say.
And to think that such bounded systems "have infinite states would thereby be capable of storing infinite information" is pseudo-mystical foolery. Large, or even "vast", is not infinite. And as sibling comment has mentioned, there is a "noise floor" below which analog resolution means nothing.