In modern deep learning there are multiple approaches that can be argued are close to some of inspirations from neuroscience: Capsule Networks, Helmholtz Machines, Energy Based Models (Score Based Generative Modeling / Diffusion Models), Associative Memory.

Information theory, Bayesian methods, Approximate Computations are more relevant for inspiration. Neuroscience is not the filed which studies intelligent behavior.

> What am I missing?

Not much!

An HTM layer requires far more parameters than a traditional deep neural network, on the order of gigabytes for one or two basic HTM layers. A cortical column is going to have many such layers, and a 1000-brains model will have many (thousands?) of cortical columns. In my opinion, the underlying idea is fantastic but the practical aspects of implementing it are daunting.

That's what killed deep learning in the 90s right? The ideas were there, they were just impractical, and now they're Python libraries on a laptop. Could be similar here albeit I don't know anything around the viability of their HTM layers!

What is "HTM layer"?

Hierarchical Temporal Memory

I know what HTM stands for. I'm asking what is "HTM layer".

An HTM layer is one feedforward layer in a Heirarchical Temporal Memory network. See for example https://arxiv.org/abs/1511.00083 which trains one-layer HTM models. See for example https://timallanwheeler.com/blog/2022/01/09/hierarchical-tem... for a more approachable blog post about the same topic

That's a very nice blog post, thanks!

I'm not sure why you think HTM layers are bigger than modern DL layers. The HTM layer configuration used in the paper (B=128, M=32, N=2048, and K=40) is 335M parameters. Compare to GPT-3 with 96 layers, where each layer has 1.8B parameters. Much larger models than GPT-3 have already appeared with no end in sight as to how much more they can scale.

The point is, if HTM worked, people would throw compute resources at it, just like they do with DL models. But it doesn't.

Yeah, Numenta is not going anywhere. DL has taken over the serious applications a while back, and Numenta doesn't have anything really substantial to offer compared to DL.

I'm actually surprised they are still around...