News for hackers. Stuff that matters.

A comment on Implementation of Google's Griffin Architecture – RNN LLM. Parent comment

Re:Implementation of Google's Griffin Architecture – RNN LLM by boywitharupee on Thursday April 11, 2024 @06:44AM (#40000711)
and is Griffin a state space model?
[ Parent | on HN ]
Re:Implementation of Google's Griffin Architecture – RNN LLM by imjonse on Thursday April 11, 2024 @08:27AM (#40001386)
No, it's a combination of RNN and Transformer.
[ Parent | on HN ]
Re:Implementation of Google's Griffin Architecture – RNN LLM by michwilinski on Thursday April 11, 2024 @08:49AM (#40001537)
I mean, SSMs are in fact under the hood RNNs
[ Parent | on HN ]
Re:Implementation of Google's Griffin Architecture – RNN LLM by VHRanger on Thursday April 11, 2024 @05:06PM (#40006800)
At the end of the day, either you carry around a hidden state, or you have a fixed window for autoregression.

You can call hidden states "RNN-like" and autoregressive windows "transformer-like", but apart from those two core paradigms I don't know of other ways to do sequence modelling.

Mamba/RWKV/Griffin are somewhere between those two extremes.

[ Parent | on HN ]