To be precise, ChatGPT 3.5 turbo being 20B is officially a mistake from a Microsoft Researcher, quoting a wrong source published before the release of chatgpt3.5 turbo. Up to you to believe it or not. But I wouldn’t claim it’s a 20B according to Microsoft Researchers.

The withdrawn paper: https://arxiv.org/abs/2310.17680

The wrong source: https://www.forbes.com/sites/forbestechcouncil/2023/02/17/is...

The discussion: https://www.reddit.com/r/LocalLLaMA/comments/17jrj82/new_mic...

It's interesting how the paper was completely retracted instead of just being corrected.

Yep. It feels like a 20B parameter model.

GPT-3 was 175B, so it'd be a bit odd if GPT-4 wasn't at least 5x larger (1T), especially since it's apparently a mixture of experts.

I think it became apparent when mixtral came out. I've noticed too during training that my model overwrites useful information so it makes sense for these types of models to have emerged.