I think the clearest evidence is Microsofts paper where they show abilities at various stages during training[1]... But in a talk [2], they give more details... The unicorn gets worse during the finetuning process.

[2]: https://www.youtube.com/watch?v=qbIk7-JPB2c&t=1392s

[1]: https://arxiv.org/abs/2303.12712

Thanks, that’s interesting.

Noobie follow up question: Should we put any trust into “Sparks of intelligence” I thought it was regarded as a Microsoft marketing piece, not a serious paper.

The data presented is true... The text might be rather exaggerated/unscientific/marketing...

Also notable that the team behind that paper wasn't involved in designing/building the model, but they did get access to prerelease versions.

I don’t trust it because enough third parties were able to verify the findings.

This is the double edge sword of being so ridiculously closed.

See my comment elsewhere on this post. Greg Brockman, head of strategic initiatives at OpenAI, was talking at a round table discussion in Korea a few weeks ago about how they had to start using the quantized (smaller, cheaper) model earlier in 2023. I noticed a switch in March 2023, with GPT-4 performance being severely degraded after that for both English-language tasks as well as code-related tasks (reading and writing).

Oh my god, this is how a lemon market[0] starts..

[0] https://en.m.wikipedia.org/wiki/The_Market_for_Lemons

ChatGPT is definitely more restricted than the API. Example:

https://news.ycombinator.com/item?id=36179783

That's disappointing, I thought ChatGPT WAS using the API. I mean what's the point of paying if you don't get similar levels of quality?

I thought that too. It's certainly how they present it. But, apparently not.

ChatGPT doesn't use the API. It uses the same underlying model with a bunch of added prompts (and possibly additional fine-tuning?) to add to make it conversational.

One would pay because what they get out of chatGPT provides value, of course. Keep in mind that the users of these 2 products can be (and in fact are) different — chatGPT is a lot friendlier (from a UX perspective) than using the API playground (or using the API itself).

They are comparing text-davinci-003 with ChatGPT which presumably uses gpt-3.5-turbo, so quite different models.

They are killing text-davinci-003 btw.

I've spent like $600 on text-davinci-003. This sucks!

We also compare ChatGPT4 vs GPT4 API in that thread and observe the same difference.