GPT-4 is getting worse over time, not better

Re:GPT-4 is getting worse over time, not better by tanaros on Wednesday July 19, 2023 @11:17AM (#36787868)

> Code generation: the change they report is that the newer GPT-4 adds non-code text to its output. They don't evaluate the correctness of the code. They merely check if the code is directly executable. So the newer model's attempt to be more helpful counted against it.

In the prompt they specifically request only the Python code, no other output. An “attempt to be helpful” that directly contradicts the user’s request seems like it should count against it.

[ Parent | on HN ]

Re:GPT-4 is getting worse over time, not better by Filligree on Wednesday July 19, 2023 @11:38AM (#36788182)

I guess, but that still isn't the sort of degradation people have been talking about. It's not a useful data point in that regard.

[ Parent | on HN ]

Re:GPT-4 is getting worse over time, not better by gwd on Wednesday July 19, 2023 @03:24PM (#36791833)

I mean, if you were hoping to use the API to generate something machine-parse-albe, and that used to work, but it doesn't any more, then sure, that's a sort of regression. But it's not a regression in coding, but a regression in following specific kinds of directions.

I certainly have found quirks like this; for instance, for a while I was asking it questions about Chinese grammar; but I wanted it only to use Chinese characters, and not to use pinyin. I tried all sorts of prompt variations to get it not to output pinyin, but was unsuccessful, and in the end gave up. But I think that's a very different class of failure than "Can't output correct code in the first place".

[ Parent | on HN ]

Re:GPT-4 is getting worse over time, not better by oefnak on Wednesday July 19, 2023 @01:41PM (#36790283)

That's false. If it outputs formatted code, it's easier to read. I don't see the backtics, I see formatted code when using the chat interface.

[ Parent | on HN ]