News for hackers. Stuff that matters.
Re:33TB of text data for a 1T-parameter model by abetusk on Saturday April 15, 2023 @11:42PM (#35587067)
By no means rigorous, an estimate for the number of neurons is around 10^10 [0] with an average number of connections estimated at around 8000 [1].

Call 10^10 \approx 2^40 for convenience, and 8000 \approx 2^13, which gives us a 2^53 entropy estimate, or about a petabyte of information as an estimate of what the human brain can store (discounting more exotic theories of memory stored in DNA or some such).

[0] https://en.wikipedia.org/wiki/Human_brain#Microanatomy

[1] https://psychology.stackexchange.com/questions/7967/how-many...

Re:33TB of text data for a 1T-parameter model by edulix on Sunday April 16, 2023 @12:48AM (#35587349)
Add to that that our neurons are more complex to point neurons used in typical Artificial neural-nets.

A single pyramid neuron in the neocortex might be more comparable to a multilayer neural net.

https://www.biorxiv.org/content/10.1101/2021.10.25.465651v1....

Re:33TB of text data for a 1T-parameter model by pmoriarty on Sunday April 16, 2023 @07:47AM (#35589162)
Their communication is much more complex too, and neither they nor their connections are static.

We don't understand how they work at the subatomic level simply because human understanding of the subatomic world is not complete, but even just at the atomic level a single neuron is massively more complex than anything humans have created.

Going up to the molecular level, even that is staggeringly more complex than the incredibly simple abstractions that make up a neural net.

Is what happens in the brain at the molecular, atomic, or subatomic levels relevant or necessary to intelligence and consciousness? We just don't know yet, but we do know all of that is far more complex and very different from the simple abstractions that are used for neural nets and LLMs.

The back of a napkin calculations in this thread don't even begin to do justice to the tremendous amount of "calculation" or "storage" that happens in the human brain.

Re:33TB of text data for a 1T-parameter model by fnordpiglet on Sunday April 16, 2023 @01:58AM (#35587650)
Although I’d note this is more like a 1pb stored model.