Re:GPT-5.6
by Wowfunhappy on Friday July 10, 2026 @07:49AM
(#48858656)
Where are they getting new data from at this point? Didn't they already read the entire internet?
[ Parent
| on HN ]
Re:GPT-5.6
by redanddead on Friday July 10, 2026 @10:09AM
(#48860192)
Custom sets and mining their own users, as every lab does
Re:GPT-5.6
by bevekspldnw on Friday July 10, 2026 @03:16PM
(#48863992)
Custom what though? And user queries are not a knowledge repository itβs an ignorance repo.
A lot of data these days is synthetically generated. As an example, to make a model good at understanding assembly you simply need to round-trip code through a compiler and disassembler, then train against the source of truth and the assembly. You can generate arbitrary algebra expressions and have it solve it.
A lot of pretraining is also choosing the right type of data, you don't want to just have it ingest garbage (although I read that some amount of garbage actually helps the model be more robust). Pretraining crystallizes a lot of the inductive biases that post-training builds on, so by crafting the right data mixture you can make it easier for it to start off with a good foundation. There is also a lot of focus on mid-training these days, which I understand is basically either the name for the synthetic data stage, or the SFT phase before all the RL