News for hackers. Stuff that matters.
Re:OpenAI dropped the price of o3 by 80% by Tiberium on Tuesday June 10, 2025 @02:40PM (#44239928)
I've heard lots of people say that, but no objective reproducible benchmarks confirm such a thing happening often. Could this simply be a case of novelty/excitement for a new model fading away as you learn more about its shortcomings?
Re:OpenAI dropped the price of o3 by 80% by Kranar on Tuesday June 10, 2025 @03:10PM (#44240288)
I used to think the models got worse over time as well but then I checked my chat history and what I noticed isn't that ChatGPT gets worse, it's that my standards and expectations increase over time.

When a new model comes out I test the waters a bit with some more ambitious queries and get impressed when it can handle them reasonably well. Over time I take it for granted and then just expect it to be able to handle ever more complex queries and get dissappointed when I hit a new limit.

Re:OpenAI dropped the price of o3 by 80% by echelon on Tuesday June 10, 2025 @03:32PM (#44240519)
Re-run your historical queries, or queries that are similarly shaped.
Re:OpenAI dropped the price of o3 by 80% by throwaway314155 on Tuesday June 10, 2025 @05:28PM (#44241648)
Sounds like a _whole_ thing.
Re:OpenAI dropped the price of o3 by 80% by sakesun on Tuesday June 10, 2025 @06:05PM (#44241977)
They could cache that :)
Re:OpenAI dropped the price of o3 by 80% by echelon on Wednesday June 11, 2025 @10:13AM (#44247853)
That would make for a very interesting timing attack.
Re:OpenAI dropped the price of o3 by 80% by herval on Tuesday June 10, 2025 @02:46PM (#44240006)
there's definitely measurements (eg https://hdsr.mitpress.mit.edu/pub/y95zitmz/release/2 ) but I imagine they're rare because those benchmarks are expensive, so nobody keeps running them all the time?

Anecdotally, it's quite clear that some models are throttled during the day (eg Claude sometimes falls back to "concise mode" - with and without a warning on the app).

You can tell if you're using Windsurf/Cursor too - there are times of the day where the models constantly fail to do tool calling, and other times they "just work" (for the same query).

Finally, there's cases where it was confirmed by the company, like Gpt-4o's sycopanth tirade that very clearly impacted its output (https://openai.com/index/sycophancy-in-gpt-4o/)

Re:OpenAI dropped the price of o3 by 80% by glitch253 on Tuesday June 10, 2025 @04:08PM (#44240928)
Cursor / Windsurf's degraded functionality is exactly why I created my own system:

https://github.com/mpfaffenberger/code_puppy

Re:OpenAI dropped the price of o3 by 80% by Deathmax on Tuesday June 10, 2025 @03:22PM (#44240417)
Your linked article is specifically comparing two different versioned snapshots of a model and not comparing the same model across time.

You've also made the mistake of conflating what's served via API platforms which are meant to be stable, and frontends which have no stability guarantees, and are very much iterated on in terms of the underlying model and system prompts. The GPT-4o sycophancy debacle was only on the specific model that's served via the ChatGPT frontend and never impacted the stable snapshots on the API.

I have never seen any sort of compelling evidence that any of the large labs tinkers with their stable, versioned model releases that are served via their API platforms.

Re:OpenAI dropped the price of o3 by 80% by herval on Tuesday June 10, 2025 @03:32PM (#44240518)
Please read it again. The article is clearly comparing gpt4 to gpt4, and gpt3.5 to gpt3.5, in march vs june 2023
Re:OpenAI dropped the price of o3 by 80% by Deathmax on Tuesday June 10, 2025 @03:38PM (#44240587)
I did read it, and I even went to their eval repo.

> At the time of writing, there are two major versions available for GPT-4 and GPT-3.5 through OpenAI’s API, one snapshotted in March 2023 and another in June 2023.

openaichat/gpt-3.5-turbo-0301 vs openaichat/gpt-3.5-turbo-0613, openaichat/gpt-4-0314 vs openaichat/gpt-4-0613. Two _distinct_ versions of the model, and not the _same_ model over time like how people like to complain that a model gets "nerfed" over time.

Re:OpenAI dropped the price of o3 by 80% by drewnick on Tuesday June 10, 2025 @03:18PM (#44240365)
I feel this too. I swear some of the coding Claude Code does on weekends is superior to the weekdays. It just has these eureka moments every now and then.
Re:OpenAI dropped the price of o3 by 80% by herval on Tuesday June 10, 2025 @03:20PM (#44240388)
Claude has been particularly bad since they released 4.0. The push to remove 3.7 from Windsurf hasn’t helped either. Pretty evident they’re trying to force people to pay for Claude Code…

Trusting these LLM providers today is as risky as trusting Facebook as a platform, when they were pushing their “opensocial” stuff

Re:OpenAI dropped the price of o3 by 80% by bobxmax on Tuesday June 10, 2025 @03:44PM (#44240657)
My suspicion is it's the personalization. Most people have things like 'memory' on, and as the models increasingly personalize towards you, that personalization is hurting quality rather than helping it.

Which is why the base model wouldn't necessarily show differences when you benchmarked them.

Re:OpenAI dropped the price of o3 by 80% by 85392_school on Tuesday June 10, 2025 @02:45PM (#44239997)
I think it's an illusion. People have been claiming it since the GPT-4 days, but nobody's ever posted any good evidence to the "model-changes" channel in Anthropic's Discord. It's probably just nostalgia.
Re:OpenAI dropped the price of o3 by 80% by cainxinth on Tuesday June 10, 2025 @04:31PM (#44241144)
I assumed it was because the first week revealed a ton of safety issues that they then "patched" by adjusting the system prompt, and thus using up more inference tokens on things other than the user's request.
Re:OpenAI dropped the price of o3 by 80% by colordrops on Tuesday June 10, 2025 @07:37PM (#44242693)
It's probably less often quantizing and more often adding more and more to their hidden system prompt to address various issues and "issues", and as we all know, adding more context sometimes has a negative effect.
Re:OpenAI dropped the price of o3 by 80% by tshaddox on Tuesday June 10, 2025 @09:15PM (#44243280)
Yeah, it’s almost certainly hallucination (by the human user).