Re:The illusion of state in state-space models
by logicchains on Friday June 07, 2024 @01:29PM
(#40610828)
If you read the rest of the paper it's clear what they mean. SSMs are theoretically no more powerful than transformers, meaning without chain of thought they can't efficiently solve state-tracking problems, unlike RNNs: https://arxiv.org/abs/2207.00729 .
[ Parent
| on HN ]
Re:The illusion of state in state-space models
by yobbo on Friday June 07, 2024 @02:27PM
(#40611466)
The difference to RNNs is that the state in SSMs is a linear combination of previous inputs, which makes it possible to parallelise training in various ways.
Gates and non-linearities in RNNs allow the state to be "any" function of previous inputs.
Re:The illusion of state in state-space models
by opprobium on Friday June 07, 2024 @02:02PM
(#40611185)
Not just efficiently, can't solve.
Re:The illusion of state in state-space models
by logicchains on Friday June 07, 2024 @02:12PM
(#40611291)
They can solve it if you keep adding layers to the transformer, it's just not efficient; you'd need exponentially more layers than a similarly sized RNN.