News for hackers. Stuff that matters.

erewhile

Re:Continuous Thought Machines on Monday May 12, 2025 @09:49AM (#43962879)
Not the original poster, but:

- Theoretical Neuroscience Computational and Mathematical Modeling of Neural Systems - Peter Dayan, L. F. Abbott (2001) is quite good, more mathematical than computational.

- Neuronal dynamics, available here: https://neuronaldynamics.epfl.ch/ is also quite good, and free to read. Has python exercises as well. If I recall correctly, it mostly goes into simulations of singular neurons, and not so much entire networks and what we can do with them, but it does a good job at bridging the chemistry / biology / math to computation.

If we're talking about papers, one I mentioned in my other comment:

- Real-Time Computing Without Stable States: A New Framework for Neural Computation Based on Perturbations, https://doi.org/10.1162/089976602760407955

- Dynamics of Sparsely Connected Networks of Excitatory and Inhibitory Spiking Neurons, by Nicolas Brunel (Don't have a DOI on hand for this one)

- Spiking Neural Networks and Their Applications: A Review, https://doi.org/10.3390/brainsci12070863 , is a very nice review of methods and does some nice explaining on concepts.

If you're looking for keywords on the topic:

- Leaky Integrate and Fire (LIF) neurons

- Spiking neural networks

- Liquid State Machines (LSM)

- Synaptic plasticity (Models of synaptic plasticity)

- Spike-based synaptic plasticity

Re:Continuous Thought Machines on Monday May 12, 2025 @04:17AM (#43960701)
The ideas of these machines isn't entirely new. There's some research from 2002, where Liquid State Machines (LSM) are introduced[1]. These are networks that generally rely on continuous inputs into spiking neural networks, which are then read by some dense layer that connects to all the neurons in this network to read what is called the liquid state.

These LSMs have also been used for other tasks, like playing Atari games in a paper from 2019[2], where they show that while sometimes these networks can outperform humans, they don't always, and they tend to fail at the same things more conventional neural networks failed at at the time as well. They don't outperform these conventional networks, though.

Honestly, I'd be excited to see more research going into continuous processing of inputs (e.g., audio) with continuous outputs, and training full spiking neural networks based on neurons on that idea. We understand some of the ideas of plasticity, and they have been applied in this kind of research, but I'm not aware of anyone creating networks like this with just the kinds of plasticity we see in the brain, with no back propagation or similar algorithms. I've tried this myself, but I think I either have a misunderstanding of how things work in our brains, or we just don't have the full picture yet.

[1] doi.org/10.1162/089976602760407955 [2] doi.org/10.3389/fnins.2019.00883