You can use tenacity for retries and wouldn't you want to cache the request / response around the endpoint instead of the gpt call -> that's what we ended up doing.

Streaming output and function-calling support is interesting

Could you elaborate on this — “wouldn’t want to cache around the endpoint instead of GPT call”. Just want to see if I’m missing an important consideration here

You're probably calling openai from a backend server. If you're making the call because a user asked a question -> your endpoint received it -> did some processing -> called openai -> returned the response,

why would you cache the openai call instead of the endpoint that's receiving the user call?

good points, probably going to add streaming output, function calling support. As for retries tenacity does a great job already