Evidence that LLMs are reaching a point of diminishing returns

Re:Evidence that LLMs are reaching a point of diminishing re... by notamy on Sunday April 14, 2024 @06:01PM (#40034853)

> Huge jump from GPT-2 to GPT-3. Huge jump from GPT-3 to GPT-4 … and not so much for GPT-4 (13 months ago) to GPT-4 Turbo (just released). It’s hard not to see this plot as tentative evidence for the hypothesis of diminishing returns. Whatever doubling there might have been, has perhaps come to an end.

Maybe I’m misunderstanding how OpenAI names their models, but was there supposed to be a huge jump from GPT-4 to GPT-4 Turbo? Just from the names it sounds like it’s expected to be in the same realm of model performance, just faster.

[ Parent | on HN ]

Re:Evidence that LLMs are reaching a point of diminishing re... by aduffy on Sunday April 14, 2024 @06:04PM (#40034883)

If OpenAI had GPT-5 they would’ve released that instead of 4-Turbo.

OpenAI was allegedly working on their Arrakis model last year that didn’t end up panning out. It’s clear they’re hitting some sort of limit in the scaling laws, or cost efficiency, or both.

[ Parent | on HN ]

Re:Evidence that LLMs are reaching a point of diminishing re... by 0x008 on Monday April 15, 2024 @06:37AM (#40038733)

Why should the release GPT-5 if GPT-4 is still at the top of all benchmarks? 95% of the population who are not techies will probably use ChatGPT anyway for everything. I think they are pretty much the leading LLM company in most ways.

It is all financially-driven. All of the AI companies are working on models constantly are are purposely putting out releases so they can be ahead of the game or have some nice news for investors in order to make sure their stocks grow.

OpenAI probably know they are ahead with GPT-5 as well and they are planning to drop it Q3/Q4 after Llama3 (which will maybe trump GPT-4), so they can be ahead again for the EOY earnings.

[ Parent | on HN ]

Re:Evidence that LLMs are reaching a point of diminishing re... by og_kalu on Monday April 15, 2024 @11:18PM (#40048055)

OpenAI released GPT 3.5 after 4 was done training so that's just wrong.

[ Parent | on HN ]

Re:Evidence that LLMs are reaching a point of diminishing re... by buildbot on Sunday April 14, 2024 @06:03PM (#40034868)

Your understanding is correct, Gary Marcus is intentionally misrepresenting what GPT-4 turbo is to make a story. It’s clearly the cost optimized version of gpt-4. That’s why it’s much cheaper per token.

[ Parent | on HN ]

Re:Evidence that LLMs are reaching a point of diminishing re... by timrobinson333 on Sunday April 14, 2024 @06:03PM (#40034873)

I think the point made in the article is that they wanted to build gpt 5 but they renamed it to 4 turbo because the improvements over 4 weren't what they were expecting

[ Parent | on HN ]

Re:Evidence that LLMs are reaching a point of diminishing re... by buildbot on Sunday April 14, 2024 @06:04PM (#40034885)

Then why is cheaper per token, and following the same path as GPT3 with several cheaper turbo versions.

[ Parent | on HN ]

Re:Evidence that LLMs are reaching a point of diminishing re... by notamy on Sunday April 14, 2024 @06:04PM (#40034880)

Right, but... is there any actual proof of that? I didn’t see any in the article.

[ Parent | on HN ]

Re:Evidence that LLMs are reaching a point of diminishing re... by aduffy on Sunday April 14, 2024 @06:06PM (#40034894)

From October: https://www.theinformation.com/articles/openai-dropped-work-...

[ Parent | on HN ]

Re:Evidence that LLMs are reaching a point of diminishing re... by buildbot on Sunday April 14, 2024 @06:08PM (#40034903)

If you read the article, the central claim is that they failed at making a more efficient model, not a larger, better, GPT-5 model.

[ Parent | on HN ]

Re:Evidence that LLMs are reaching a point of diminishing re... by firebaze on Sunday April 14, 2024 @06:04PM (#40034878)

The author suggests that GPT4-Turbo should have been GPT-5, but turned out too weak. Not sure if I buy that personally, I'm quite confident that OpenAI can deliver a lot more than GPT4 does right now, just not in a way that is economically feasible. GPT4-Turbo may be an intermediate release to have something to show while more work is being done behind the scenes.

[ Parent | on HN ]