News for hackers. Stuff that matters.
Re:OpenAI starts testing ChatGPT premium by shrimpx on Friday January 13, 2023 @04:19PM (#34374019)
Makes sense. Do you know if there exits a re-implementation of a ChatGPT-like stateful conversational interface on top of the GPT-3 API? A cursory search doesn't turn up anything.
Re:OpenAI starts testing ChatGPT premium by shagie on Friday January 13, 2023 @04:30PM (#34374154)
If you look at

    https://beta.openai.com/playground/p/default-chat?model=text-davinci-003
    https://beta.openai.com/playground/p/default-friend-chat?model=text-davinci-003
    https://beta.openai.com/playground/p/default-marv-sarcastic-chat?model=text-davinci-003
you can see a chat "session".

The issue is that to maintain the state, you need to maintain the history of the conversation.

For example, the friend chat starts out at 28 request tokens. I submit it, get a response back, and it's at 45 tokens. I add in a bit of my own commentary... and it's 59 tokens. I hit submit and its 80 tokens.... and so on.

https://imgur.com/a/wJ7LNAf

That session wasn't particularly insightful, but you get the idea.

The issue is that I did three requests with 28 + 59 + 88 tokens. As I keep going, this can add up. To maintain that context, my previous history (and the chat response) remains in the prompt for the next message growing arithmetically.

---

(edit)

Working from the chat instead of friend chat... a bit more interesting conversation.

https://imgur.com/Dog2jOT

Note however, that at this point the next request will be at least 267 tokens. We're still talking fractions of a penny ($0.002 / 1k tokens) - but we are talking money that I can look at rather than even smaller fractions. And as noted - it grows each time you continue with the conversation.

Re:OpenAI starts testing ChatGPT premium by shrimpx on Saturday January 14, 2023 @05:14AM (#34378998)
Thank you, that's really insightful! Presumably you could get it to summarize the conversation when the conversation reaches max tokens.
Re:OpenAI starts testing ChatGPT premium by shagie on Saturday January 14, 2023 @11:48AM (#34381640)
Likely yes. You could even switch to a less intensive model for doing that (e.g. Curie). The Curie model is 1/10th as costly as Davinci to run. Running 8k tokens through Davinci is $0.16 while Curie would only be $0.016 - and that's likely showing up in back end compute and should be considered if someone was building their own chat bot on top of gpt3.