Re:OpenAI starts testing ChatGPT premium
by shrimpx on Saturday January 14, 2023 @05:14AM
(#34378998)
Thank you, that's really insightful! Presumably you could get it to summarize the conversation when the conversation reaches max tokens.
Re:OpenAI starts testing ChatGPT premium
by shagie on Saturday January 14, 2023 @11:48AM
(#34381640)
Likely yes. You could even switch to a less intensive model for doing that (e.g. Curie). The Curie model is 1/10th as costly as Davinci to run. Running 8k tokens through Davinci is $0.16 while Curie would only be $0.016 - and that's likely showing up in back end compute and should be considered if someone was building their own chat bot on top of gpt3.
The issue is that to maintain the state, you need to maintain the history of the conversation.
For example, the friend chat starts out at 28 request tokens. I submit it, get a response back, and it's at 45 tokens. I add in a bit of my own commentary... and it's 59 tokens. I hit submit and its 80 tokens.... and so on.
https://imgur.com/a/wJ7LNAf
That session wasn't particularly insightful, but you get the idea.
The issue is that I did three requests with 28 + 59 + 88 tokens. As I keep going, this can add up. To maintain that context, my previous history (and the chat response) remains in the prompt for the next message growing arithmetically.
---
(edit)
Working from the chat instead of friend chat... a bit more interesting conversation.
https://imgur.com/Dog2jOT
Note however, that at this point the next request will be at least 267 tokens. We're still talking fractions of a penny ($0.002 / 1k tokens) - but we are talking money that I can look at rather than even smaller fractions. And as noted - it grows each time you continue with the conversation.