News for hackers. Stuff that matters.
Re:GPT-6 Astra, Looped Transformers, and Hidden Reasoning by cainxinth on Wednesday September 09, 2026 @01:10PM (#49629808)
It's the same story every time OpenAI or Anthropic releases a new model. They are generous with compute for the first few days, and use maximum fidelity with uncompressed weights. Everything runs at its best to make a good first impression. But eventually they pare things back and the models perform a little worse.
Re:GPT-6 Astra, Looped Transformers, and Hidden Reasoning by Vetch on Wednesday September 09, 2026 @03:00PM (#49632196)
The most charitable explanation I can think of for this is something like regression to the mean. When a model is first released, there'll be a subset of users who, just by chance, sample the highest quality band of the distribution that answers their query. Some of them will rush over to social media and post about how amazing a model is. Over time, those users' mental model of responses will converge but they'll perceive the model's return to typical performance as a downgrade.

This guess/explanation predicts that most users won't match what the initial social media hype claims, doesn't discount user experience as simple habituation nor does it assume companies are lying when they say there have been no changes to the model itself (quantization included).

I also think there's an aspect where initial testing is more forgiving because the more persnickety polish bits can be ignored and tests are likely to have similar structure to things that can be trained for. Meanwhile, actual specific work items are a broader unusual distribution with more stringent acceptance criteria.

Personally, I can detect a separation between Sol and Astra (but not as large as that between Opus and Fable). While they can solve most of the same problems, Astra takes less time, is less frustrating to talk to, is cleaner, notices more, spins wheels less and requires less corrections.

Re:GPT-6 Astra, Looped Transformers, and Hidden Reasoning by kmeisthax on Wednesday September 09, 2026 @05:04PM (#49634304)
To add onto this, if you use a shiny new model and it gives you a turd, you're not going to tweet about it ("hey guys, look what I made with Astra! Nothing!"), and even if you do nobody is going to interact with it so it does poorly in the algorithm, because it has to compete with all the people using the new model to make something that looks impressive. Then people get tired of the magic trick and the logic flips.
Re:GPT-6 Astra, Looped Transformers, and Hidden Reasoning by zaphirplane on Wednesday September 09, 2026 @06:34PM (#49635541)
Really? there would be complaints, it’s expensive and doesn’t do as well
Re:GPT-6 Astra, Looped Transformers, and Hidden Reasoning by foolswisdom on Wednesday September 09, 2026 @08:52PM (#49636808)
When it's happened to me, I shrugged and went back to the way I did things before. Then again, I'm not a vocal social media user by any means.
Re:GPT-6 Astra, Looped Transformers, and Hidden Reasoning by majormajor on Thursday September 10, 2026 @12:47AM (#49638533)
We've seen that some---gpt5 was considered pretty lackluster intially, in particular. Opus 4.7 and 5 vs 4.6 were also greeted with a lot more "meh" than 4.6 or Fable.
Re:GPT-6 Astra, Looped Transformers, and Hidden Reasoning by roywiggins on Wednesday September 09, 2026 @06:34PM (#49635543)
Another effect may be that with newer models, people try out the hard problems they got stuck with in older models, and when that model happens to succeed, they are very impressed.

Perhaps the new model really is better... but perhaps merely being prompted to try again with the difficult problem just gave them a new dice roll and they came up lucky. And then regression to the mean kicks in.

Re:GPT-6 Astra, Looped Transformers, and Hidden Reasoning by majormajor on Thursday September 10, 2026 @12:46AM (#49638527)
> I also think there's an aspect where initial testing is more forgiving because the more persnickety polish bits can be ignored and tests are likely to have similar structure to things that can be trained for. Meanwhile, actual specific work items are a broader unusual distribution with more stringent acceptance criteria.

IMO this is 90% of it (as someone who has a bit of a different interaction style and runs these things less autonomously, and hasn't generally seen the claimed regressions). Day 1: throw new stuff at it that failed badly, exciting to see something make more progress! Day n: reality sets in that it still wasn't perfect the first time.

Re:GPT-6 Astra, Looped Transformers, and Hidden Reasoning by dooglius on Wednesday September 09, 2026 @02:35PM (#49631527)
Do you have hard evidence of this assertion?
Re:GPT-6 Astra, Looped Transformers, and Hidden Reasoning by simlevesque on Wednesday September 09, 2026 @02:47PM (#49631881)
We can't have hard evidence. It's a SaaS and they own the code and the machine it runs on.

So it may be a widespread hallucination. But there's no evidence of that either.

Re:GPT-6 Astra, Looped Transformers, and Hidden Reasoning by fragmede on Wednesday September 09, 2026 @03:13PM (#49632471)
We could still have soft evidence though. Make a Todo app on Monday, and make a Todo app on Tuesday, and see what it makes in comparison.
Re:GPT-6 Astra, Looped Transformers, and Hidden Reasoning by ArvidSu on Wednesday September 09, 2026 @03:45PM (#49633045)
You only need to come up with a catchy "SomethingBench" name, post it on reddit/x and now you're an ai sage. Not to disparage the launch/after comparison though, I'd genuinely enjoy a data point like that
Re:GPT-6 Astra, Looped Transformers, and Hidden Reasoning by marcus_cemes on Wednesday September 09, 2026 @03:41PM (#49632974)
You would need a significant sample size to make any sort of conclusion from such a probabilistic process. Then there's the issue of how you would actually grade/compare.
Re:GPT-6 Astra, Looped Transformers, and Hidden Reasoning by luckydata on Wednesday September 09, 2026 @03:50PM (#49633132)
someone already does that https://aistupidlevel.info/
Re:GPT-6 Astra, Looped Transformers, and Hidden Reasoning by chaimtweiss on Wednesday September 09, 2026 @04:48PM (#49634038)
It's actually a extremely cool site, and fascinating to view the results off the AI bots i use.
Re:GPT-6 Astra, Looped Transformers, and Hidden Reasoning by dooglius on Wednesday September 09, 2026 @06:10PM (#49635212)
Run a benchmark with a large number of samples, rerun a few days later. Compare results, use statistics to see if there's a statistically significant difference.
Re:GPT-6 Astra, Looped Transformers, and Hidden Reasoning by kadoban on Thursday September 10, 2026 @12:20AM (#49638362)
Doing this in a way that doesn't get you noticed or fucked with is potentially going to be quite difficult.

If they have a "hey we're being benchmarked" mode, which is not hard to imagine, avoiding tripping it is going to be annoying and difficult to prove.

Re:GPT-6 Astra, Looped Transformers, and Hidden Reasoning by dooglius on Thursday September 10, 2026 @08:27AM (#49642651)
You would want to use your own benchmark, certainly, not something publicly known. But outside of that I don't think "this is a benchmark" is such an easy category to determine -- a benchmark should be similar to a typical scenario anyway.
Re:GPT-6 Astra, Looped Transformers, and Hidden Reasoning by kadoban on Thursday September 10, 2026 @09:25AM (#49643354)
> "this is a benchmark" is such an easy category to determine

I mean, it's not _that_ hard to determine most likely, and/or it's hard to be sure you didn't get found out by llm-assisted analysis on your traffic. It's not going to be a one-shot request and response it's going to be a whole bunch of them in an artificial way, by nature.

And then anything you found is single-use only if you're paranoid because even if they find out later, they have your benchmark now (because you sent it to them to use it even if you don't publish it).

Re:GPT-6 Astra, Looped Transformers, and Hidden Reasoning by bradly on Wednesday September 09, 2026 @02:56PM (#49632111)
There is a toot from an Open AI person a couple days ago saying they are "pulling all the levers" because of capacity issues. I have no idea what the heck the person is talking about, but I'm guessing there are consequence for those levers.

    > "Demand for Astra is really unprecedented. We're pulling all the levers possible to sustain the demand, but I've not seen anything like it until now and we went through very steep growth before. Priority will always be to keep excellent service for existing users, but we might have to pause new Pro subscriptions for a bit if this continues."
Re:GPT-6 Astra, Looped Transformers, and Hidden Reasoning by nonethewiser on Wednesday September 09, 2026 @03:35PM (#49632869)
Depends on the nature of the levers
Re:GPT-6 Astra, Looped Transformers, and Hidden Reasoning by baby on Wednesday September 09, 2026 @01:39PM (#49630250)
You think they introduce stronger quantization after a few days?
Re:GPT-6 Astra, Looped Transformers, and Hidden Reasoning by boredatoms on Wednesday September 09, 2026 @01:50PM (#49630447)
For sure they quickly move to q8, the output quality difference to bf16 is small compared to the speed/capacity gain
Re:GPT-6 Astra, Looped Transformers, and Hidden Reasoning by torginus on Wednesday September 09, 2026 @04:02PM (#49633351)
Some people here have remarked previously that while reduced precision doesn't show up in quick prompts, it does severely impact these models' ability to perform long running tasks - to the point that running these big models with severe quantization might be counterproductive as smaller but less quantized ones perform better.
Re:GPT-6 Astra, Looped Transformers, and Hidden Reasoning by NineStarPoint on Wednesday September 09, 2026 @02:23PM (#49631165)
Yeah q8 made so littler difference back when I was testing such things I'd be surprised if people could quickly notice that as a change. It's got to be either further quantized or some other type of optimization that kicks in when people notice the drop.
Re:GPT-6 Astra, Looped Transformers, and Hidden Reasoning by selectodude on Wednesday September 09, 2026 @03:11PM (#49632434)
NVFP4 would buy them a huge increase in capacity but I think it would be noticeable.
Re:GPT-6 Astra, Looped Transformers, and Hidden Reasoning by Caracas288 on Wednesday September 09, 2026 @03:55PM (#49633225)
Why doesn't someone just try to measure this next time!?
Re:GPT-6 Astra, Looped Transformers, and Hidden Reasoning by embedding-shape on Wednesday September 09, 2026 @06:33PM (#49635527)
Can't really measure without being sure you aren't being messed around with, when it's a remote platform. Stupidly easy to detect when people run such benchmarks/tests against you as well.
Re:GPT-6 Astra, Looped Transformers, and Hidden Reasoning by nonethewiser on Wednesday September 09, 2026 @03:33PM (#49632850)
Could this explain Opus?
Re:GPT-6 Astra, Looped Transformers, and Hidden Reasoning by blurbleblurble on Wednesday September 09, 2026 @01:26PM (#49630057)
Or a lot worse
Re:GPT-6 Astra, Looped Transformers, and Hidden Reasoning by holler on Wednesday September 09, 2026 @01:41PM (#49630283)
so, AGI is cancelled?
Re:GPT-6 Astra, Looped Transformers, and Hidden Reasoning by __MatrixMan__ on Wednesday September 09, 2026 @01:43PM (#49630313)
AGI for the peasants is cancelled.
Re:GPT-6 Astra, Looped Transformers, and Hidden Reasoning by elwell on Wednesday September 09, 2026 @05:32PM (#49634708)
Trogdor - the AGInator
Re:GPT-6 Astra, Looped Transformers, and Hidden Reasoning by holoduke on Wednesday September 09, 2026 @06:44PM (#49635659)
I am sure every input send to openai is prechecked by a dumb model and then send to another one. They heavily tweak this to improve performance.
Re:GPT-6 Astra, Looped Transformers, and Hidden Reasoning by OneOffAsk on Wednesday September 09, 2026 @10:33PM (#49637654)
It’s all speculation (you too), but I think the effect you’re describing is instead getting calibrated to the model’s limits. Next time a new model comes out, wait a month before trying and see if you have the same feeling of rapid quality decline after a few days. I did after I jumped back into it mid 5.x or whatever ChatGPT after paternity leave. Blown away for a few days, worried about my job for a few days, then increasingly aware of its limits.
Re:GPT-6 Astra, Looped Transformers, and Hidden Reasoning by cbg0 on Thursday September 10, 2026 @07:51AM (#49642190)
This is a bit of an urban myth. There are trackers which keep historical performance and Sol hasn't been nerfed: https://marginlab.ai/trackers/codex/
Re:GPT-6 Astra, Looped Transformers, and Hidden Reasoning by webern777 on Saturday September 12, 2026 @05:11AM (#49670504)
The problem with this is I won't get any clicks posting that performance is the same and I don't let facts get in the way of more clicks.