Re:GPT-6 Astra, Looped Transformers, and Hidden Reasoning
by fragmede on Wednesday September 09, 2026 @03:13PM
(#49632471)
We could still have soft evidence though. Make a Todo app on Monday, and make a Todo app on Tuesday, and see what it makes in comparison.
Re:GPT-6 Astra, Looped Transformers, and Hidden Reasoning
by ArvidSu on Wednesday September 09, 2026 @03:45PM
(#49633045)
You only need to come up with a catchy "SomethingBench" name, post it on reddit/x and now you're an ai sage. Not to disparage the launch/after comparison though, I'd genuinely enjoy a data point like that
Re:GPT-6 Astra, Looped Transformers, and Hidden Reasoning
by marcus_cemes on Wednesday September 09, 2026 @03:41PM
(#49632974)
You would need a significant sample size to make any sort of conclusion from such a probabilistic process. Then there's the issue of how you would actually grade/compare.
Re:GPT-6 Astra, Looped Transformers, and Hidden Reasoning
by luckydata on Wednesday September 09, 2026 @03:50PM
(#49633132)
someone already does that https://aistupidlevel.info/
Re:GPT-6 Astra, Looped Transformers, and Hidden Reasoning
by chaimtweiss on Wednesday September 09, 2026 @04:48PM
(#49634038)
It's actually a extremely cool site, and fascinating to view the results off the AI bots i use.
Re:GPT-6 Astra, Looped Transformers, and Hidden Reasoning
by dooglius on Wednesday September 09, 2026 @06:10PM
(#49635212)
Run a benchmark with a large number of samples, rerun a few days later. Compare results, use statistics to see if there's a statistically significant difference.
Re:GPT-6 Astra, Looped Transformers, and Hidden Reasoning
by kadoban on Thursday September 10, 2026 @12:20AM
(#49638362)
Doing this in a way that doesn't get you noticed or fucked with is potentially going to be quite difficult.
If they have a "hey we're being benchmarked" mode, which is not hard to imagine, avoiding tripping it is going to be annoying and difficult to prove.
Re:GPT-6 Astra, Looped Transformers, and Hidden Reasoning
by dooglius on Thursday September 10, 2026 @08:27AM
(#49642651)
You would want to use your own benchmark, certainly, not something publicly known. But outside of that I don't think "this is a benchmark" is such an easy category to determine -- a benchmark should be similar to a typical scenario anyway.
Re:GPT-6 Astra, Looped Transformers, and Hidden Reasoning
by kadoban on Thursday September 10, 2026 @09:25AM
(#49643354)
> "this is a benchmark" is such an easy category to determine
I mean, it's not _that_ hard to determine most likely, and/or it's hard to be sure you didn't get found out by llm-assisted analysis on your traffic. It's not going to be a one-shot request and response it's going to be a whole bunch of them in an artificial way, by nature.
And then anything you found is single-use only if you're paranoid because even if they find out later, they have your benchmark now (because you sent it to them to use it even if you don't publish it).
So it may be a widespread hallucination. But there's no evidence of that either.