News for hackers. Stuff that matters.
Re:OpenAI dropped the price of o3 by 80% by lispisok on Tuesday June 10, 2025 @02:15PM (#44239674)
I swear every time a new model is released it's great at first but then performance gets worse over time. I figured they were fine-tuning it to get rid of bad output which also nerfed the really good output. Now I'm wondering if they were quantizing it.
Re:OpenAI dropped the price of o3 by 80% by Tiberium on Tuesday June 10, 2025 @02:40PM (#44239928)
I've heard lots of people say that, but no objective reproducible benchmarks confirm such a thing happening often. Could this simply be a case of novelty/excitement for a new model fading away as you learn more about its shortcomings?
Re:OpenAI dropped the price of o3 by 80% by Kranar on Tuesday June 10, 2025 @03:10PM (#44240288)
I used to think the models got worse over time as well but then I checked my chat history and what I noticed isn't that ChatGPT gets worse, it's that my standards and expectations increase over time.

When a new model comes out I test the waters a bit with some more ambitious queries and get impressed when it can handle them reasonably well. Over time I take it for granted and then just expect it to be able to handle ever more complex queries and get dissappointed when I hit a new limit.

Re:OpenAI dropped the price of o3 by 80% by echelon on Tuesday June 10, 2025 @03:32PM (#44240519)
Re-run your historical queries, or queries that are similarly shaped.
Re:OpenAI dropped the price of o3 by 80% by throwaway314155 on Tuesday June 10, 2025 @05:28PM (#44241648)
Sounds like a _whole_ thing.
Re:OpenAI dropped the price of o3 by 80% by sakesun on Tuesday June 10, 2025 @06:05PM (#44241977)
They could cache that :)
Re:OpenAI dropped the price of o3 by 80% by echelon on Wednesday June 11, 2025 @10:13AM (#44247853)
That would make for a very interesting timing attack.
Re:OpenAI dropped the price of o3 by 80% by herval on Tuesday June 10, 2025 @02:46PM (#44240006)
there's definitely measurements (eg https://hdsr.mitpress.mit.edu/pub/y95zitmz/release/2 ) but I imagine they're rare because those benchmarks are expensive, so nobody keeps running them all the time?

Anecdotally, it's quite clear that some models are throttled during the day (eg Claude sometimes falls back to "concise mode" - with and without a warning on the app).

You can tell if you're using Windsurf/Cursor too - there are times of the day where the models constantly fail to do tool calling, and other times they "just work" (for the same query).

Finally, there's cases where it was confirmed by the company, like Gpt-4o's sycopanth tirade that very clearly impacted its output (https://openai.com/index/sycophancy-in-gpt-4o/)

Re:OpenAI dropped the price of o3 by 80% by glitch253 on Tuesday June 10, 2025 @04:08PM (#44240928)
Cursor / Windsurf's degraded functionality is exactly why I created my own system:

https://github.com/mpfaffenberger/code_puppy

Re:OpenAI dropped the price of o3 by 80% by Deathmax on Tuesday June 10, 2025 @03:22PM (#44240417)
Your linked article is specifically comparing two different versioned snapshots of a model and not comparing the same model across time.

You've also made the mistake of conflating what's served via API platforms which are meant to be stable, and frontends which have no stability guarantees, and are very much iterated on in terms of the underlying model and system prompts. The GPT-4o sycophancy debacle was only on the specific model that's served via the ChatGPT frontend and never impacted the stable snapshots on the API.

I have never seen any sort of compelling evidence that any of the large labs tinkers with their stable, versioned model releases that are served via their API platforms.

Re:OpenAI dropped the price of o3 by 80% by herval on Tuesday June 10, 2025 @03:32PM (#44240518)
Please read it again. The article is clearly comparing gpt4 to gpt4, and gpt3.5 to gpt3.5, in march vs june 2023
Re:OpenAI dropped the price of o3 by 80% by Deathmax on Tuesday June 10, 2025 @03:38PM (#44240587)
I did read it, and I even went to their eval repo.

> At the time of writing, there are two major versions available for GPT-4 and GPT-3.5 through OpenAI’s API, one snapshotted in March 2023 and another in June 2023.

openaichat/gpt-3.5-turbo-0301 vs openaichat/gpt-3.5-turbo-0613, openaichat/gpt-4-0314 vs openaichat/gpt-4-0613. Two _distinct_ versions of the model, and not the _same_ model over time like how people like to complain that a model gets "nerfed" over time.

Re:OpenAI dropped the price of o3 by 80% by drewnick on Tuesday June 10, 2025 @03:18PM (#44240365)
I feel this too. I swear some of the coding Claude Code does on weekends is superior to the weekdays. It just has these eureka moments every now and then.
Re:OpenAI dropped the price of o3 by 80% by herval on Tuesday June 10, 2025 @03:20PM (#44240388)
Claude has been particularly bad since they released 4.0. The push to remove 3.7 from Windsurf hasn’t helped either. Pretty evident they’re trying to force people to pay for Claude Code…

Trusting these LLM providers today is as risky as trusting Facebook as a platform, when they were pushing their “opensocial” stuff

Re:OpenAI dropped the price of o3 by 80% by bobxmax on Tuesday June 10, 2025 @03:44PM (#44240657)
My suspicion is it's the personalization. Most people have things like 'memory' on, and as the models increasingly personalize towards you, that personalization is hurting quality rather than helping it.

Which is why the base model wouldn't necessarily show differences when you benchmarked them.

Re:OpenAI dropped the price of o3 by 80% by 85392_school on Tuesday June 10, 2025 @02:45PM (#44239997)
I think it's an illusion. People have been claiming it since the GPT-4 days, but nobody's ever posted any good evidence to the "model-changes" channel in Anthropic's Discord. It's probably just nostalgia.
Re:OpenAI dropped the price of o3 by 80% by cainxinth on Tuesday June 10, 2025 @04:31PM (#44241144)
I assumed it was because the first week revealed a ton of safety issues that they then "patched" by adjusting the system prompt, and thus using up more inference tokens on things other than the user's request.
Re:OpenAI dropped the price of o3 by 80% by colordrops on Tuesday June 10, 2025 @07:37PM (#44242693)
It's probably less often quantizing and more often adding more and more to their hidden system prompt to address various issues and "issues", and as we all know, adding more context sometimes has a negative effect.
Re:OpenAI dropped the price of o3 by 80% by tshaddox on Tuesday June 10, 2025 @09:15PM (#44243280)
Yeah, it’s almost certainly hallucination (by the human user).
Re:OpenAI dropped the price of o3 by 80% by nabla9 on Tuesday June 10, 2025 @02:26PM (#44239775)
It seems that least Google is overselling their compute capacity.

You pay monthly fee, but Gemini is completely jammed 5-6 hours when North America is working.

Re:OpenAI dropped the price of o3 by 80% by baq on Tuesday June 10, 2025 @02:44PM (#44239985)
Gemini is simply that good. I’m trying out Claude 4 every now and then and go back to Gemini to fix its mess…
Re:OpenAI dropped the price of o3 by 80% by energy123 on Tuesday June 10, 2025 @03:19PM (#44240387)
Gemini is the best model in the world. Gemini is the worst web app in the world. Somehow those two things are coexisting. The web devs in their UI team have really betrayed the hard work of their ML and hardware colleagues. I don't say this lightly - I say this after having paid attention to critical bugs, more than I can count on one hand, that persisted for over a year. They either don't care or are grossly incompetent.
Re:OpenAI dropped the price of o3 by 80% by thorum on Tuesday June 10, 2025 @03:41PM (#44240611)
Try AI Studio if you haven’t already: https://aistudio.google.com/
Re:OpenAI dropped the price of o3 by 80% by koakuma-chan on Tuesday June 10, 2025 @04:14PM (#44240990)
Re:OpenAI dropped the price of o3 by 80% by nabla9 on Tuesday June 10, 2025 @04:25PM (#44241094)
Well said.

Google is best in pure AI research, both quality and volume. They have sucked at productization for years. Not not just AI but other products as well. Real mystery.

Re:OpenAI dropped the price of o3 by 80% by energy123 on Tuesday June 10, 2025 @04:36PM (#44241191)
I don't understand why they can't just make it fast and go through the bug reports from a year ago and fix them. Is it that hard to build a box for users to type text into without it lagging for 5 seconds or throwing a bunch of errors?
Re:OpenAI dropped the price of o3 by 80% by baq on Thursday June 12, 2025 @02:35AM (#44254805)
If it doesn’t make sense, it makes sense. Nobody will get their promo by ‘fixing bugs’.
Re:OpenAI dropped the price of o3 by 80% by fasterthanlime on Tuesday June 10, 2025 @02:54PM (#44240098)
Funny, I have the exact opposite experience! I use Claude to fix Gemini’s mess.
Re:OpenAI dropped the price of o3 by 80% by symfoniq on Tuesday June 10, 2025 @03:00PM (#44240156)
Maybe LLMs just make messes.
Re:OpenAI dropped the price of o3 by 80% by hgomersall on Tuesday June 10, 2025 @02:58PM (#44240134)
I heard that, but I'm getting consistent garbage from Gemini.
Re:OpenAI dropped the price of o3 by 80% by dayjah on Tuesday June 10, 2025 @03:18PM (#44240379)
For code? Use the context7 mcp.
Re:OpenAI dropped the price of o3 by 80% by edzitron on Tuesday June 10, 2025 @04:32PM (#44241152)
When you say "jammed," how do you mean?
Re:OpenAI dropped the price of o3 by 80% by JamesBarney on Tuesday June 10, 2025 @02:53PM (#44240087)
I'm pretty sure this is just a psychological phenomenon. When a new model is released all the capabilities the new model has that the old model lacks are very salient. This makes it seem amazing. Then you get used to the model, push it to the frontier, and suddenly the most salient memories of the new model are it's failures.

There are tons of benchmarks that don't show any regressions. Even small and unpublished ones rarely show regressions.

Re:OpenAI dropped the price of o3 by 80% by mhitza on Tuesday June 10, 2025 @02:32PM (#44239843)
That was my suspicion when I first deleted my account, when it felt the output got worse in ChatGPT and I found highly suspicious when I saw an errand davinci model keyword in the chatgpt url.

Now I'm feeling similarly with their image generation (which is the only reason I created a paid account two months ago, and the output looks more generic by default).

Re:OpenAI dropped the price of o3 by 80% by beering on Tuesday June 10, 2025 @09:01PM (#44243206)
Are you able to quantify how quickly your perception gets skewed by how long you use the models?
Re:OpenAI dropped the price of o3 by 80% by mhitza on Wednesday June 11, 2025 @01:23PM (#44249782)
I can't quantity it for my past experience, that was more than a year ago, and I wasn't using ChatGPT daily at the time either.

This time around it felt pretty stark. I used ChatGPT to create at most 20 different image compositions. And after a couple of good ones at first, it felt worse after. One thing I've noticed recently is that when working on vector art compositions, the results start more simplistic, and often enough look like clipart thrown together. This wasn't my experience first time around. Might be temperature tweaks, or changes in their prompt that lead to this effect. Might be some random seed data they use, who knows.

Re:OpenAI dropped the price of o3 by 80% by codr7 on Tuesday June 10, 2025 @02:21PM (#44239726)
If they could, they would.

All the proof I need.

Re:OpenAI dropped the price of o3 by 80% by daseiner1 on Tuesday June 10, 2025 @02:24PM (#44239753)
It's still a very competitive marketplace
Re:OpenAI dropped the price of o3 by 80% by mathgradthrow on Tuesday June 10, 2025 @02:40PM (#44239940)
honestly refreshing take.
Re:OpenAI dropped the price of o3 by 80% by bboygravity on Tuesday June 10, 2025 @02:34PM (#44239871)
But OpenAI breathes honesty. They're open source! They would never do such a thing. /s
Re:OpenAI dropped the price of o3 by 80% by solfox on Tuesday June 10, 2025 @02:32PM (#44239838)
I have seen this behavior as well.
Re:OpenAI dropped the price of o3 by 80% by JoshuaDavid on Tuesday June 10, 2025 @08:33PM (#44243047)
I suspect what's happening is that lots of people have a collection of questions / private evals that they've been testing on every new model, and when a new model comes out it sometimes can answer a question that previous models couldn't. So that selects for questions where the new model is at the edge of its capabilities and probably got lucky. But when you come up with a new question, it's generally going to be on the level of the questions the new model is newly able to solve.

Like I suspect if there was a "new" model which was best-of-256 sampling of gpt-3.5-turbo that too would seem like a really exciting model for the first little bit after it came out, because it could probably solve a lot of problems current top models struggle with (which people would notice immediately) while failing to do lots of things that are a breeze for top models (which would take people a little bit to notice).

Re:OpenAI dropped the price of o3 by 80% by beering on Tuesday June 10, 2025 @09:00PM (#44243202)
It’s easy to measure the models getting worse, so you should be suspicious that nobody who claims this has scientific evidence to back it up.