News for hackers. Stuff that matters.
Re:Continuous Thought Machines by iandanforth on Monday May 12, 2025 @05:24AM (#43961112)
This paper is concerning. While divorced from the standard ML literature there is a lot of work on biologically plausible spiking, timing dependant artificial neutral networks. The nomenclature here doesn't seem to acknowledge that body of work. Instead it appears as a step toward that bulk of research coming from the ML/LLM field without a clear appreciation of the ground well traveled there.*

In addition some of the terminology is likely to cause confusion. By calling a synaptic integration step "thinking" the authors are going to confuse a lot of people. Instead of the process of forming an idea, evaluating that idea, potentially modifying it and repeating (what a layman would call thinking) they are trying to ascribe "thinking" to single unit processes! That's a pretty radical departure from both ML and ANN literature. Pattern recognition/signal discrimination is well known at the level of synaptic integration and firing, but "thinking?" No, that wording is not helpful.

*I have not reviewed all the citations and am reacting to the plain language of the text as someone familiar with both lines of research.

Re:Continuous Thought Machines by cepera on Monday May 12, 2025 @09:13AM (#43962541)
>there is a lot of work on biologically plausible spiking

I ask you kindly to share the list (or even better brief review) of most insightful books/papers in your opinion with neuroscience inspired algorithms concepts/implementation details.

Re:Continuous Thought Machines by erewhile on Monday May 12, 2025 @09:49AM (#43962879)
Not the original poster, but:

- Theoretical Neuroscience Computational and Mathematical Modeling of Neural Systems - Peter Dayan, L. F. Abbott (2001) is quite good, more mathematical than computational.

- Neuronal dynamics, available here: https://neuronaldynamics.epfl.ch/ is also quite good, and free to read. Has python exercises as well. If I recall correctly, it mostly goes into simulations of singular neurons, and not so much entire networks and what we can do with them, but it does a good job at bridging the chemistry / biology / math to computation.

If we're talking about papers, one I mentioned in my other comment:

- Real-Time Computing Without Stable States: A New Framework for Neural Computation Based on Perturbations, https://doi.org/10.1162/089976602760407955

- Dynamics of Sparsely Connected Networks of Excitatory and Inhibitory Spiking Neurons, by Nicolas Brunel (Don't have a DOI on hand for this one)

- Spiking Neural Networks and Their Applications: A Review, https://doi.org/10.3390/brainsci12070863 , is a very nice review of methods and does some nice explaining on concepts.

If you're looking for keywords on the topic:

- Leaky Integrate and Fire (LIF) neurons

- Spiking neural networks

- Liquid State Machines (LSM)

- Synaptic plasticity (Models of synaptic plasticity)

- Spike-based synaptic plasticity

Re:Continuous Thought Machines by rkp8000 on Monday May 12, 2025 @12:25PM (#43964784)
A (non-exhaustive) list of some notable papers:

Maass 2002, Real-time computing without stable states: https://pubmed.ncbi.nlm.nih.gov/12433288/

Sussillo & Abbott 2009, Generating Coherent Patterns of Activity from Chaotic Neural Networks https://pmc.ncbi.nlm.nih.gov/articles/PMC2756108/

Abbott et al 2016, Building functional networks of spiking model neurons https://pubmed.ncbi.nlm.nih.gov/26906501/

Zenke & Ganguli 2018, SuperSpike: Supervised Learning in Multilayer Spiking Neural Networks https://ganguli-gang.stanford.edu/pdf/17.superspike.pdf

Bellec et al 2020, A solution to the learning dilemma for recurrent networks of spiking neurons https://www.nature.com/articles/s41467-020-17236-y

Payeur et al 2021, Burst-dependent synaptic plasticity can coordinate learning in hierarchical circuits https://www.nature.com/articles/s41593-021-00857-x

Cimesa et al 2023, Geometry of population activity in spiking networks with low-rank structure https://journals.plos.org/ploscompbiol/article?id=10.1371/jo...

Ororbia 2024, Contrastive signal–dependent plasticity: Self-supervised learning in spiking neural circuits https://www.science.org/doi/10.1126/sciadv.adn6076 Kudithipudi et al 2025, Neuromorphic computing at scale (review) https://www.nature.com/articles/s41586-024-08253-8

Re:Continuous Thought Machines by tiahura on Monday May 12, 2025 @09:34AM (#43962717)
The authors don't label a single synaptic integration as "thinking." They use the term for the network-wide internal loop ("internal ticks") that unrolls after every external input, and explicitly say it is merely "analogous to thought."
Re:Continuous Thought Machines by vonneumannstan on Monday May 12, 2025 @10:52AM (#43963724)
Was this written by Jürgen Schmidhuber?
Re:Continuous Thought Machines by mountainriver on Monday May 12, 2025 @02:27PM (#43966101)
Agree, they are presenting this like its a new idea without hardly any reference to the decades of work on spiking neural nets and similar.
Re:Continuous Thought Machines by TeMPOraL on Monday May 12, 2025 @03:16PM (#43966566)
I'm sort of not surprised; my impression is that, for the past decade or two, ML researchers who did acknowledge related work in neuroscience were broadly accused of hubris for daring to compare their work to biological brains.
Re:Continuous Thought Machines by program_whiz on Monday May 12, 2025 @05:19PM (#43967556)
Sorry I should have responded to this comment, but I wrote a separate response in the parent thread. I didn't feel the pdf / paper was really trying to mimick spiking biological networks in all but the loosest sense (there is a sequence of activations and layers of "neurons"). I think the major contribution is just using the dot product on output transpose output, the rest is just diffusion / attention on inputs. Its conceptually a combination of "input attention" and "output attention" using a kind of stepped recursive model.