News for hackers. Stuff that matters.

qwelfkh

Re:Markov Chains Are the Original Language Models on Tuesday September 23, 2025 @04:21PM (#45352257)
Okay, so we have a function that converts a buffer of tokens to a distribution of next tokens. We give the function a buffer, get back the next-token-distribution, from which we sample to get our next token. We append that to the buffer (and possibly roll off the first token) to obtain a new buffer. If we consider entire buffers to be states (which is a perfectly reasonable thing to do), then we have a stochastic process that moves us from one state to another. How is that not a Markov chain?