"Why are ChatGPT responses written a few words at a time?"
The responses are broken up into smaller chunks for several reasons:
1. Better readability: Breaking up the responses into smaller pieces makes it easier for users to read and understand the information being provided.
2. Interactivity: ChatGPT's response format allows for a back-and-forth interaction with the user, making it feel more like a conversation and allowing the user to ask follow-up questions or clarify information.
3. Error correction: By responding a few words at a time, the model can quickly correct any errors or misunderstandings in real-time, leading to a more accurate overall conversation.
4. Efficient communication: Responding in smaller chunks allows the model to quickly provide information and reduces the latency in the communication process.
Overall, the goal is to create an efficient, user-friendly, and interactive conversational experience.
For those who don't know, it is like Markov Chains: probability of next word (or a group of words encoded as a token) is calculated based on previous words and computationally intensive. It just uses not just probability between 1-2 previous tokens, it uses 2048 token window (roughly 1500 words) to predict next token, then puts next token into window and goes on.