33TB of text data for a 1T-parameter model

Re:33TB of text data for a 1T-parameter model by neatze on Saturday April 15, 2023 @09:01PM (#35586259)

Brain is nothing like attention networks at least in terms of dynamics and individual neuron complexity, my wild ass guess; there is not enough space in DNA to store even 10% of brain connections.

What weights are you referring to ?

[ Parent | on HN ]

Re:33TB of text data for a 1T-parameter model by BobbyJo on Saturday April 15, 2023 @09:08PM (#35586308)

Cluster density, cluster location, eagerness to connect, distribution of cell types, etc. are all coded in DNA and contribute to creating "initial weights". Neuron "weights" need not be stored completely uncompressed.

Obviously newborns need to develop and take in stimulus before they "know" anything, so the initial conditions are not sufficient, but they are obviously necessary to make the limited learning useful.

[ Parent | on HN ]

Re:33TB of text data for a 1T-parameter model by neatze on Sunday April 16, 2023 @04:07AM (#35588150)

It always seems naive to compare neural structures and functions to artificial networks, dynamics are simply not there, even single neural would require multi layered network (if not mistaken 7 layer network, and very long training time), it is fascinating how brain self-(re)organizes through out life time from small number of examples here and there.

As far my understanding goes to very large degree it is unknown how to model such dynamics, where it would be possible to start with no spiking neurons and evolve effective/stable learning behavior from small number of examples/experiences.

[ Parent | on HN ]

Re:33TB of text data for a 1T-parameter model by stevenhuang on Monday April 17, 2023 @04:15AM (#35598040)

For all we know it may not even be necessary to model the higher fidelity aspects of our brains for a form of intelligence to emerge.

There may be many roads to intelligence. The path biology and evolution took may just be one such path.

[ Parent | on HN ]

Re:33TB of text data for a 1T-parameter model by neatze on Monday April 17, 2023 @07:06AM (#35599126)

I accept this possibility, but because we don't understand, seems wrong to assume any other path, furthermore, there is some evidence that intelligence/cognition is convergent in neural structure and function, in addition to that most ANN are inspired from studying real neurons.

[ Parent | on HN ]

Re:33TB of text data for a 1T-parameter model by pmoriarty on Sunday April 16, 2023 @08:04AM (#35589253)

"Cluster density, cluster location, eagerness to connect, distribution of cell types, etc. are all coded in DNA and contribute to creating "initial weights""

Are they coded in DNA, or do epigenetic factors, the environment cells grow in, adjacent cells, their interaction, etc, play a role here as well?

[ Parent | on HN ]

Re:33TB of text data for a 1T-parameter model by tomohelix on Saturday April 15, 2023 @09:28PM (#35586410)

That would depend on what you mean by "connection". The human genome has approximately 200GB of data in it. And this data is highly compressed as it takes 3 different layers of decompression before it gets to usable form.

[ Parent | on HN ]

Re:33TB of text data for a 1T-parameter model by le-mark on Saturday April 15, 2023 @10:25PM (#35586679)

> decompression

Does this refer to DNA -> MRNA -> to protein or is there some other mechanism here?

[ Parent | on HN ]

Re:33TB of text data for a 1T-parameter model by tomohelix on Saturday April 15, 2023 @10:37PM (#35586740)

Post translation modifications are another layer.

Also, since you asked, I would like to mention protein interactions with cellular components and tissues. These can be argued as just proteins interacting with each other but the extreme complexity of these higher levels and the unique phenotypes arising from them make me think they are deserved to be treated as another layer of "data decompression".

[ Parent | on HN ]

Re:33TB of text data for a 1T-parameter model by fnordpiglet on Sunday April 16, 2023 @02:14AM (#35587712)

A fruit fly has no meaningful learned intelligence yet is more capable than any trained neural net. It’s just a small bundle of nerves laid out according to the model encoded in their DNA. It was trained through millions of years of genetic algorithms, and is fully encoded in the DNA, along with the entire ability to specify the entire physical structure of its body. Why is it surprising our brain has instinctual ability encoded in its structure stemming from our DNA, evolved over the history of life?

[ Parent | on HN ]

Re:33TB of text data for a 1T-parameter model by bobbylarrybobby on Sunday April 16, 2023 @02:39AM (#35587811)

Not sure about the 10% point. You could imagine that we got “lucky” and a handful of simple proteins manage to produce the brain (or at least enough of the brain’s connections that the rest can be filled in during infancy via sensory input). Or put another way, maybe evolution optimized for brains that are incredibly adept while only taking a small number of codons (relative to the number of neurons) to describe. The fact that we've managed to l describe the universe through a small set of laws shows that the laws of the universe require much less space to encode than the universe itself, so it's theoretically possible.

[ Parent | on HN ]

Re:33TB of text data for a 1T-parameter model by mensetmanusman on Sunday April 16, 2023 @11:35AM (#35590894)

We have managed to describe a small subset of the universe through a small set of laws. The current incompatibility of gravity and QM and the resulting dark energy/matter explanation means we can’t see or interact with most of the universe.

[ Parent | on HN ]