It might be calculating the next word, because you can only write one word at a time, but you can't say that the current next word isn't influenced by a few words ahead. Don't think the current understanding of what LLM's do internally can rule that out.
Occasionally the act of writing out an idea in words clarifies or changes my mind about that idea, causing me to edit or rewrite what I’ve already written.
And if you believe you do that subconsciously, then how can you be sure you don’t subconsciously plan a few words ahead?
But it doesn’t write better as in has new, challenging ideas. Or ways to move/relate to humans on a personal level.
GPT writes clear, concise, authoritative, boring, and generic text.
Shaping your words to sound correct is very common. Both in speech and in writing. Sometimes it is finding how to fit a word you want to use into a sentence. Sometimes it is building a rhyme.
You may feel that you go a word at a time, but that really shows how embedded language is.
In their calculating the 'next' word, as part of that 'weighting', are the 'simple rules' for future words, that you are saying humans do but LLM's can't.
Anyway, point is 80% of this comment was already in my head before I started typing; the other 20% was light editing and an H2G2 reference.