Re:Markov Chains Are the Original Language Models
by hiddencost on Thursday February 01, 2024 @11:32AM
(#39217753)
The whole point of Transformers is that they broke the Markov assumption (i.e., that the next token probability is strictly conditioned on a window of N preceding tokens).
[ Parent
| on HN ]
Re:Markov Chains Are the Original Language Models
by pclmulqdq on Thursday February 01, 2024 @11:38AM
(#39217870)
That's not actually true, they still have a fixed history window. The idea that transformers capture through the attention mechanism is that not all past tokens are created equal, and that the importance of tokens in that history window depends on what they are.
Re:Markov Chains Are the Original Language Models
by sudosysgen on Thursday February 01, 2024 @12:07PM
(#39218261)
They also scale differently - Markov Chains scale exponentially with the size of the window, while transformers scale quadratically. So in fact transformers are really more exponentially more efficient, though without bound on resources their capabilities are a strict subset of that of Markov chain.