News for hackers. Stuff that matters.
Re:Quadrillion-parameter transformers and how to train them by spwa4 on Sunday November 08, 2026 @10:56AM (#50414325)
Aside from violating everything we know about neural networks (technically this would mean there is a way to reduce the rank of even something like ChatGPT to 1. There isn't. No way) it would also be VERY unlike what DNA does. DNA has lots of ways to express things that aren't easily disturbed by mistakes. Those mistakes are then an actual positive, in the very long term, because they help natural selection along.

A random seed that works would be the exact opposite of that.

Re:Quadrillion-parameter transformers and how to train them by nico on Sunday November 08, 2026 @05:36PM (#50418186)
> Aside from violating everything we know about neural networks

Not sure exactly what this means

But there’s prior art on the concept of compressing models through lossy representations

In particular, procedural generation of models, check out HyperNEAT (2009), HyperNetworks (2017) and Developmental Symbolic Encoding (2011)

Re:Quadrillion-parameter transformers and how to train them by spwa4 on Monday November 09, 2026 @10:48AM (#50427188)
> Not sure exactly what this means

It means that neural networks are a mistake, that they don't really have to be as complex as they are and animal minds could just calculate directly from the seed as opposed to building and powering this very expensive organ. The brain uses more power than any muscle in essentially all mammals and even more in humans. Only olympic-level efforts come anywhere close to power usage like the brain on a single muscle. So your body is really giving up something serious to let you think, to keep track of 100 Trillion weights and constantly evaluate against them.

This would be something very strange for DNA/evolution to do. Especially where it comes to energy use, if it can be done with 10W of power, you can bet your firstborn DNA will NEVER under any circumstances get caught using 10.01W.