News for hackers. Stuff that matters.
Re:The illusion of state in state-space models by logicchains on Friday June 07, 2024 @01:29PM (#40610828)
If you read the rest of the paper it's clear what they mean. SSMs are theoretically no more powerful than transformers, meaning without chain of thought they can't efficiently solve state-tracking problems, unlike RNNs: https://arxiv.org/abs/2207.00729 .
Re:The illusion of state in state-space models by yobbo on Friday June 07, 2024 @02:27PM (#40611466)
The difference to RNNs is that the state in SSMs is a linear combination of previous inputs, which makes it possible to parallelise training in various ways.

Gates and non-linearities in RNNs allow the state to be "any" function of previous inputs.

Re:The illusion of state in state-space models by opprobium on Friday June 07, 2024 @02:02PM (#40611185)
Not just efficiently, can't solve.
Re:The illusion of state in state-space models by logicchains on Friday June 07, 2024 @02:12PM (#40611291)
They can solve it if you keep adding layers to the transformer, it's just not efficient; you'd need exponentially more layers than a similarly sized RNN.