Well, it is a Markov chain if you do greedy sampling, which 99% of the time you do. So the weird part is why it still works so well.

If you do beam search, RAG, tool usage, etc then the whole system no longer is one.