> gpt4 >> gpt3.5

Leftwise bitshift would make gpt4 worse though :P

Do you have a non-trivial example of this?

I tried to make GPT-4 create an algorithm for an enhanced tic-tac-toe game (think Wordle vs Quordle to somewhat visualize the difference) and it's failing miserably.

https://arxiv.org/abs/2303.12712 there are a lot of examples in this paper.

Also it’s not that it writes perfect code straight away and you never have to re-prompt or change anything manually. But it’s still an insane speed-up.

Do you know if there are any papers published by people who don't work at Microsoft so may actually be more objective?

Just try it yourself, it’s not that hard…

Welcome to the age of the unbenchmarkable product. You say one thing, someone says another, and here we are...

There are programming benchmarks though (data contamination issues left aside)

Yes, which is funny because there was an article on here with some pretty hard data which showed not much difference or maybe worse performance from GPT4 than 3.5-turbo.

You'll likely refute that as your mind is already made up, but there you go, another conflicting and confusing data point.

What are talking about? Just compare the output of a 3.5 vs 4 yourself for a problem you are interested in, it’s a single click in the interface.. Do you always need a study or an “expert“ to make up your mind?

Benchmarks are good here because you might be a less experienced programmer than others (or maybe more experienced?), then you will tell me it's "insane bro", but that's only a matter of perspective. A benchmark gives us facts, not opinions.

I'm sure ChatGPT4 would likely agree :)

"Also gpt4 >> gpt3.5 for coding. "

This is not my subjective experience. It's better, sometimes somewhat, but also much slower. I'm not sure which one is the better tradeoff for real work. I have it on 4 by default now, don't want to think about this for every question I ask, but for my work, 3.5 was fine.