:-)
My code does retry and the entire application is written to detect and work around breakage. But eventually I do need to get enough content from OpenAI API to be able to make progress, and I am not.
At the moment, for example, all requests just time out after 12 minutes (on my side). No amount of "fixing my code" will help, and I don't want OpenAI to hold my hand, I just want it to a) return some data at least sometimes, b) not charge me for data not delivered.
Let's look at my billing page: over the last hour it shows 8 requests. A total of 52584 tokens. Not a single response made it back to me.
We spend over $20k/mo with them and don't have any issues like this.
12 minute timeouts make no sense because first of all why are you even waiting 12 minutes?
Get off HN and go fix your shit
For example exponential back off is the first step, then adding retrying on timeouts (we use streaming and if there are 30 seconds in between getting data back we retry the whole request - rare but happens), then fixing anything else that pops up
It is possible to have a stable production app on top of it
Just fix your code and stop expecting OpenAI to hold your hand