News for hackers. Stuff that matters.

A comment on Scaling Transformer to 1M tokens and beyond with RMT. Parent comment

Re:Scaling Transformer to 1M tokens and beyond with RMT by pclmulqdq on Monday April 24, 2023 @09:47AM (#35687594)
It sounds like that's a good use for encoder-decoder architecture. GPT-3 and LLaMA are decoder-only architectures.

Training that encoder sounds hard, though.

[ Parent | on HN ]

There are no comments on this comment yet.