News for hackers. Stuff that matters.
Re:Large Language Models: Scaling Laws and Emergent Properties by SideburnsOfDoom on Friday April 21, 2023 @02:02PM (#35657646)
> Mm not sure that bound applies because neurons are neither of fixed size nor fixed energy

Neurons are not of unbounded size nor energy.

Re:Large Language Models: Scaling Laws and Emergent Properties by hooloovoo_zoo on Friday April 21, 2023 @02:41PM (#35658141)
What’s the bound?
Re:Large Language Models: Scaling Laws and Emergent Properties by SideburnsOfDoom on Friday April 21, 2023 @04:43PM (#35659807)
Can you not work this out yourself? Use some common sense. Do you disagree, are you saying that animal body cells can be infinitely large?
Re:Large Language Models: Scaling Laws and Emergent Properties by hooloovoo_zoo on Friday April 21, 2023 @06:30PM (#35661110)
No bound, got it.
Re:Large Language Models: Scaling Laws and Emergent Properties by SideburnsOfDoom on Friday April 21, 2023 @06:47PM (#35661275)
Which animal body cells can be infinitely large then? This is foolery. "Neurons are of unbounded size" is a ridiculous statement, we any fool can see it's false. IDK what you're aiming at here, try making a case for something sensible?
Re:Large Language Models: Scaling Laws and Emergent Properties by hooloovoo_zoo on Friday April 21, 2023 @07:05PM (#35661437)
We hit comment cooldown but it's not foolery exactly, more like reflecting the futility of trying to make these kinds of arguments. The problem with this whole chain is we haven't actually defined what a neuron is. Take the purported bound above. Suppose I move an associated potassium ion farther away. Have I not changed the state? If yes, what's the limit to how far away I can move that ion and have it still count as part of the neuron? That's just one of the limits of physical laws, you can make very precise statements, but only under very narrow constraints.
Re:Large Language Models: Scaling Laws and Emergent Properties by SideburnsOfDoom on Saturday April 22, 2023 @03:58AM (#35664335)
Here are some bounds, from most theoretical to most practical:

The size of the earth's biosphere

The size of the largest animal.

The size of a human skull cavity.

The size of a human skull cavity, divided by the number of neurons therein ( a few Billion).

Is this not obvious? There's nothing futile about stating these obvious facts.

Do not fool around with "if I move a potassium ion farther away, isn't that a different bound" ? correct, but moving the bound slightly, is irrelevant to the obvious statement that there are still bounds, and they are practical.

Stating that the neuron is not well-defined is similar irrelevance, if you want to precisely measure the bound, you will need to know precisely what is bounded, but I did not, merely stated the obvious truth that every such definition will come with a bound. One can say "definitely smaller than this" in a wide range of ways, without defining the exact edge.

What's the alternate argument: "I won't define it, therefor it might be larger than the solar system". This is not a serious thing to say.

And to think that such bounded systems "have infinite states would thereby be capable of storing infinite information" is pseudo-mystical foolery. Large, or even "vast", is not infinite. And as sibling comment has mentioned, there is a "noise floor" below which analog resolution means nothing.