Runs on cheaper but slower compute maybe? Given all the hype and little competition, I'm sure they're willing to make it slower if it reduces cost.

Also, it is in their best interest to pick the low hanging fruit little by little so that they can continue releasing a new versions of chatgpt every six months. I think they will only accelerate the progress of chatgpt when their moat of first move advantage is being assaulted. There are five types of moats: Low-cost production; High switching costs; Network effects; Intangible assets; Efficient scale.

I disagree. It does much, much better on selected tasks. I cannot quite figure out how to describe what the difference "feels" like, but the performance is sometimes markedly different when feeding ChatGPT-3.5 and ChatGPT-4 the same prompt.

One task that ChatGPT-3.5 is hilariously bad at is reversing strings (both words and pseudorandom input). It seems to have only a vague concept of what that means, even if I try to hold its hand through the process. Maybe some prompt engineering can get it to succeed on anything longer than four letters.

ChatGPT-4 meanwhile seems to have no issue with this at all.

Have you tried inserting spaces between the characters? This may just be a tokenization issue, rather than anything due to the model per se.

Reversing a string is somewhat of a pathological case for language models, because they see tokens not characters. Learning that the token “got” and token “tog” are mirror images is only useful for string reversal and generating palindromes. Unless they are trained specifically for this task, they may not be able to do it. They should however be able to see that “g o t” and “t o g” are mirror images.

Infamously, early versions of GPT-3 tokenized numbers as grouped tokens, nerfing its calculation abilities, because it would tokenize a number such as 12345 as (illustratively) 12 34 5 which is obviously a harmful representation.

> What could be the reason for that other than much larger parameter count?

Longer inference time... I should have written it down now that people are asking about it, but a few weeks ago I was seeing people discuss the GPT-4 "paper" in what little information was released and that throwing more inference compute at the problem gives better responses.

>, 4 seems a little better than 3.5 but not by a huge amount.

Can you define that in a tangible way? I don't think most of us can since we have so little access to the product.

> 4 seems a little better than 3.5 but not by a huge amount.

Depends on the task. 3.5 was completely incapable of doing math, but 4 seems to be able to at a solid highschool graduate level.