News for hackers. Stuff that matters.
Re:GPT-5.6 by 10xDev on Thursday July 09, 2026 @01:56PM (#48849934)
Seeing the dramatic differences in scores just going from high to xhigh is just another demonstration of the bitter lesson: Just keep scaling search and learning. We are probably going to need a lot more GPUs.
Re:GPT-5.6 by energy123 on Thursday July 09, 2026 @04:18PM (#48851823)
> Dramatic difference

Isn't this just the difference between getting 0 right and getting 1 right?

Re:GPT-5.6 by bevekspldnw on Thursday July 09, 2026 @02:43PM (#48850580)
These aren’t raw base models they are the result of a ton of RLHF and various adjustments.

Bitter lesson wildly overstated in this context.

Re:GPT-5.6 by froh on Thursday July 09, 2026 @03:38PM (#48851339)
rlhf = reinforcement learning from human feedback

(had to look it up)

Re:GPT-5.6 by visarga on Friday July 10, 2026 @09:17AM (#48859525)
I think it's more RLVR (reinforcement learning from verified rewards). The RLHF is just to align models to human preferences, meaning to behave nice.
Re:GPT-5.6 by redanddead on Friday July 10, 2026 @10:07AM (#48860158)
What makes you say that
Re:GPT-5.6 by versteegen on Friday July 10, 2026 @10:52PM (#48868213)
More accurate to say RLHF aligns models to human preferences, most significantly to be helpful.
Re:GPT-5.6 by fastball on Thursday July 09, 2026 @04:00PM (#48851627)
More RLHF is in fact scaling.
Re:GPT-5.6 by bevekspldnw on Thursday July 09, 2026 @05:09PM (#48852411)
Yes, but not in the “dump another chunk of all written language in the bucket and stir”-sense which is what bitter lesson became synonymous with.

That may not be the intent of the original article, but over the past few years that’s what the phrase turned into.

Re:GPT-5.6 by avarun on Thursday July 09, 2026 @05:59PM (#48852932)
GPT-6 is supposed to be using a much larger base model that just finished pretraining so the "dump another chunk of all written language" approach is still going strong.
Re:GPT-5.6 by linkregister on Thursday July 09, 2026 @07:34PM (#48853855)
Modern pretraining also consists of expensive human-led specialized task creation and grading loops. Synthetic generation and distillation from previous models is another input for training. I wonder how much new text contributes beyond keeping knowledge up-to-date.
Re:GPT-5.6 by zamadatix on Thursday July 09, 2026 @10:42PM (#48855143)
The measure is to see if the results scale, not just the rumored attempts at building such a model, as o3 taught us.
Re:GPT-5.6 by Wowfunhappy on Friday July 10, 2026 @07:49AM (#48858656)
Where are they getting new data from at this point? Didn't they already read the entire internet?
Re:GPT-5.6 by redanddead on Friday July 10, 2026 @10:09AM (#48860192)
Custom sets and mining their own users, as every lab does
Re:GPT-5.6 by bevekspldnw on Friday July 10, 2026 @03:16PM (#48863992)
Custom what though? And user queries are not a knowledge repository it’s an ignorance repo.
Re:GPT-5.6 by krackers on Friday July 10, 2026 @04:53PM (#48865100)
A lot of data these days is synthetically generated. As an example, to make a model good at understanding assembly you simply need to round-trip code through a compiler and disassembler, then train against the source of truth and the assembly. You can generate arbitrary algebra expressions and have it solve it.

A lot of pretraining is also choosing the right type of data, you don't want to just have it ingest garbage (although I read that some amount of garbage actually helps the model be more robust). Pretraining crystallizes a lot of the inductive biases that post-training builds on, so by crafting the right data mixture you can make it easier for it to start off with a good foundation. There is also a lot of focus on mid-training these days, which I understand is basically either the name for the synthetic data stage, or the SFT phase before all the RL

Re:GPT-5.6 by bevekspldnw on Friday July 10, 2026 @03:14PM (#48863971)
Where did they get a second copy of all books, written/audio/visual media, and the web?

Couch cushions?

Re:GPT-5.6 by theptip on Thursday July 09, 2026 @10:53PM (#48855212)
The bitter lesson just means “compute scaling beats hand-tuned architectures in the long run”.

As GP said. More RLHF is in fact the bitter lesson.

Re:GPT-5.6 by cma on Thursday July 09, 2026 @04:11PM (#48851746)
The scaling with reasoning models is more and more with things like verifiable rewards (coding and math), in line with bitter lesson and also Sutton invented lots of modern RL.
Re:GPT-5.6 by theptip on Thursday July 09, 2026 @10:51PM (#48855202)
Nah, the last few generations have more RLVR in the data mix. Which is more CPU intensive and very much amenable to the bitter lesson as you can reduce the loss by doing more rollouts in your tool environment.
Re:GPT-5.6 by chaos_emergent on Thursday July 09, 2026 @11:11PM (#48855314)
RLHF is an increasingly small part of training though? From what I understand most of the capability gain is in RLVR
Re:GPT-5.6 by snthpy on Friday July 10, 2026 @01:36AM (#48856106)
Not just RLHF but also RLVR, and isn't that the litter lesson though?

My sense of the Sutton Dwarkesh interview was that he was calling out that he didn't mean just longer datasets, but rather learning through exploration and that's exactly RL.

Re:GPT-5.6 by andai on Friday July 10, 2026 @02:20AM (#48856391)
They just need more contact with reality. That's what RL is right? Contact with narrow subsets of reality.
Re:GPT-5.6 by 10xDev on Friday July 10, 2026 @09:17AM (#48859523)
Not sure where I implied they are "raw base models" and not sure what "various adjustments" means here or how "ton of RLHF" contradicts anything. If we look at research for open source models, "adjustments" usually come in the form of efficiency gains which directly contributes to the ability to scale or synthetic data pipelines to increase the dataset and increasing the context window.
Re:GPT-5.6 by altcognito on Thursday July 09, 2026 @02:04PM (#48850063)
While I think this is true, remember as we get more efficient we just decide to scale even bigger. So more GPUs, and more efficient.

I agree with the sibling comment, effiency is probably the more important component at this point. We are hitting not just a practical engineering roadblock for scaling with current technology, I think we have definitely hit a financial and logistical roadblock for up scaling with the number of GPUs (on an immediate basis)

Re:GPT-5.6 by Salgat on Thursday July 09, 2026 @02:54PM (#48850717)
Kind of refreshing though that the "throw more processing at it" scaling we saw in the 90s has returned in a different way. For a while we were really bottlenecked in our advances by relatively low levels of parallelism (most software used by your average user doesn't scale cleanly with more than a few threads).
Re:GPT-5.6 by vatsachak on Thursday July 09, 2026 @02:00PM (#48849993)
I mean, theoretically you can solve every finitary problem with a brute force solution...

Richard Sutton specifically states that the search has to be smart. We know that the brain uses recurrent connections and is shallow. I think a lot more money has to go into architecture. Feed Forward transformers can only scale so far

Re:GPT-5.6 by dyauspitr on Thursday July 09, 2026 @04:00PM (#48851624)
And dozens of data centers in every state so tokens are dirt cheap.
Re:GPT-5.6 by emp17344 on Thursday July 09, 2026 @02:34PM (#48850479)
This isn’t really how it works anymore. Agents rely heavily on tool use and the agentic harness to perform tasks. Pre-training is no longer very effective.
Re:GPT-5.6 by HDThoreaun on Thursday July 09, 2026 @02:43PM (#48850583)
I thought models werent allowed tools on arc-agi?
Re:GPT-5.6 by Razengan on Thursday July 09, 2026 @02:15PM (#48850214)
> We are probably going to need a lot more GPUs.

Or a breakthrough in algorithms etc.

The human brain, heck all bio brains, are proof that you don't need a lot of power or size for intelligence.

Re:GPT-5.6 by ryandvm on Thursday July 09, 2026 @03:11PM (#48850965)
The human brain has 80 billion neurons and a 100 trillion synapses. I think you're underselling the processing power of that warm chunk of meat.

The real message of the last 15 years has actually been the opposite: if you throw enough processing power at it, intelligence emerges.

Re:GPT-5.6 by dbspin on Thursday July 09, 2026 @03:32PM (#48851261)
Moreover we've known for quite a while now that glial cells also participate in cognition and moderate learning (e.g.: [1]). When you take those connections into account the numbers get really staggering. 85 billion glial cells with trillions of protein channels facilitating communication between the glial syncytium [2].

[1] https://www.sciencedirect.com/science/article/pii/S193459091... [2] https://pmc.ncbi.nlm.nih.gov/articles/PMC5063692/

Re:GPT-5.6 by orbital-decay on Thursday July 09, 2026 @04:55PM (#48852242)
The real question is not how many "weights" the human brain has (neurons+synapses may or may not translate into "weights", and brain might be also inefficient for what it is), but rather how much evolutionary and social "compute" was necessary to pack everything into that capacity.
Re:GPT-5.6 by thunky on Thursday July 09, 2026 @08:04PM (#48854151)
I think you're helping GPs point: there is a lot of efficiency gains to be made to match the processing power of the brain, given it's size and power draw.
Re:GPT-5.6 by baq on Friday July 10, 2026 @02:19AM (#48856385)
It’s still 20W. We have living proof what is possible within 20W. The message has always been clear - try to get silicon computer to be as power efficient as the brain is as it is obviously possible.
Re:GPT-5.6 by torginus on Friday July 10, 2026 @06:50AM (#48858230)
I keep thinking about the cyborg chicken mechs in Metal Gear Solid 4. If stuff like Neuralink gets advanced enough, we might find that cutting up animals and sticking electrodes in their brains is the best way to make robots.

Yeah, people might object, but it can be argued that we are already subjecting scores of animals to horrors beyond comprehension just to get a bucket of chicken wings. And even if we manage to get silicon to do what brains do, it will likely cost 1000x as much and consume 1000x the power like you said.

It's hell of an economic incentive.

Re:GPT-5.6 by altcognito on Thursday July 09, 2026 @02:27PM (#48850377)
20 watts for inference AND training!
Re:GPT-5.6 by aeyes on Thursday July 09, 2026 @02:34PM (#48850482)
For intelligence, I expect the next breakthrough to be colocation of memory and compute in the same chip. And we'll need much more of this memory, probably a few petabytes.
Re:GPT-5.6 by navigate8310 on Thursday July 09, 2026 @04:24PM (#48851898)
There goes my plan to buy a PC for the next decade
Re:GPT-5.6 by echelon on Thursday July 09, 2026 @07:38PM (#48853901)
The whole of knowledge work is being automated. We've barely begun to see the GPU build out. This is just the start.

I'd imagine they're going to 10x this, maybe 100x this.

Re:GPT-5.6 by y1n0 on Thursday July 09, 2026 @09:44PM (#48854776)
yeah, spacex's planned million satellite datacenter-in-space constellation doesn't seem as absurd.
Re:GPT-5.6 by Schlagbohrer on Friday July 10, 2026 @04:08AM (#48857137)
Sam Altman was saying we'll eventually need to build a dyson sphere. We'll see how far human society can withstand this kind of escalation. I'm guessing one more year before there are major fractures in the basic economic/social/political relationship that get so bad it actually prevents more buildout.
[dead]
Re:GPT-5.6 by versteegen on Friday July 10, 2026 @10:58PM (#48868260)
I agree. But I think you're missing that LLMs can internalise a lot of the thinking process in their layers without explicit CoT. That System 1-style reasoning is bounded depth computation but very, very broad. Yudkowsky called it "cached thoughts" and I think it's an incredibly important idea [1]. It's really stiking how the best LLMs don't even need to think where smaller LLMs do.

So as more thinking is cached in their weights through increased RL training, those weights are doing more useful work and the efficiency is increasing.

[1] https://www.lesswrong.com/posts/2MD3NMLBPCqPfnfre/cached-tho...

Re:GPT-5.6 by orbital-decay on Thursday July 09, 2026 @04:48PM (#48852173)
Or a lot better efficiency.
Re:GPT-5.6 by dmitriy_ko on Thursday July 09, 2026 @05:29PM (#48852598)
And a lot more electricity to power them.
Re:GPT-5.6 by XCSme on Thursday July 09, 2026 @05:38PM (#48852702)
Not always, in some cases, changing to a higher reasoning makes the AI doubt itself too much, and skip over the correct answer by overcomplicating the problem and polluting the context.

It would be nice to see on which categories of problems the extra thinking makes it better and on which it makes it worse.

Re:GPT-5.6 by andai on Friday July 10, 2026 @02:22AM (#48856414)
I think I have this problem but with my human brain.
Re:GPT-5.6 by Schlagbohrer on Friday July 10, 2026 @04:09AM (#48857142)
This shows up in OpenAI's graphs on their announcement page. There is a peak performance datapoint in the graphs past which (to the right on the graph indicating more resources spent) peformance declines. And it's on every graph on that page!
Re:GPT-5.6 by XCSme on Friday July 10, 2026 @04:26AM (#48857269)
And in my tests, that point of "overthinking" depends on the problem's complexity, so it's not necessarily that using "xhigh" is always bad or good.
Re:GPT-5.6 by hyperbovine on Thursday July 09, 2026 @09:33PM (#48854709)
Or a new model. The human brain does far more with far less.
Re:GPT-5.6 by ld4nt3 on Friday July 10, 2026 @05:01AM (#48857514)
Yes an no current models can read and output much more faster with differing quality tho.
Re:GPT-5.6 by andai on Friday July 10, 2026 @02:19AM (#48856381)
I said a few months ago, "man, Opus is great, but sometimes when talking with it I have the feeling like, this thing should be about 10 times bigger."

When Mythos was announced after that, I was pleasantly surprised to hear about it. But when it turned out to be only two times bigger, I was a little disappointed!

(I am even more disappointed with the safety filters, but that's kind of a separate discussion... "Fortunately" I find that I can usually edit my prompt by single character and get through...)

Re:GPT-5.6 by Schlagbohrer on Friday July 10, 2026 @04:10AM (#48857153)
What do you mean bigger? Bigger functional context window?
Re:GPT-5.6 by nbardy on Friday July 10, 2026 @10:51AM (#48860801)
We’re definitely going to need a lot of Gpu’s