If you define the state broadly enough to include the state of the entire machine itself, including all of its internal representations, weights, activations, etc, then you are playing a funny trick here.

By the same reasoning, a human brain is also Markov process.

What you are doing here is a vast oversimplification and it is practically useless for understanding how LLMs work.