News for hackers. Stuff that matters.
Re:The Kimi K3 Moment by milkshakes on Saturday July 18, 2026 @03:54PM (#48961676)
assume you are a "second class lab" and you are in fact making progress by distilling the results of the frontier labs' efforts.

what is the end game for this strategy?

if the frontier labs shut down, or stop releasing to the public, and there's noting left to distill, how will you progress?

Re:The Kimi K3 Moment by InsideOutSanta on Saturday July 18, 2026 @04:12PM (#48961855)
This line of thinking makes no sense because it assumes that labs that distill from frontier models are doing nothing else. It's the classic "the Chinese can only copy" mentality, and it's going to end poorly for American companies.

I'm pretty sure that all labs are distilling each others' LLMs, maybe apart from Anthropic and OpenAI. It would be stupid not to do it, because it's cheap and effective. But that's not the only thing they're doing. If you think K3 and GLM-5.2 got this good only from distilling frontier models, you're not paying attention to Chinese labs' publications.

Re:The Kimi K3 Moment by milkshakes on Saturday July 18, 2026 @04:39PM (#48962141)
i never assumed that, and i do keep up with the publications. i'm also not saying it's a dumb thing to do! what i am saying is that empirically, it appears that distillation of a more advanced model is a required first step for them to train a borderline competitive, cheaper model. in effect, their training is subsidized by the frontier labs.

if this were not the case, then we would be observing chinese models that far surpass frontier models in capabilities, rather than "almost as good, but much cheaper", and we would be having a very different conversation. what happens to these efforts when the subsidy is cut off?

Re:The Kimi K3 Moment by InsideOutSanta on Saturday July 18, 2026 @04:50PM (#48962247)
> empirically, it appears that distillation of a more advanced model is a required first step

I see no evidence for that.

> if this were not the case, then we would be observing chinese models that far surpass frontier models

It's pretty clear that the primary reason for the difference is budget and compute availability. Chinese labs have at least an order of magnitude less money than Anthropic and OpenAI.

> what happens to these efforts when the subsidy is cut off?

They will continue making progress as they do now, minus the benefits of distillation.

Re:The Kimi K3 Moment by milkshakes on Saturday July 18, 2026 @05:15PM (#48962496)
https://www.anthropic.com/news/detecting-and-preventing-dist...

Moonshot AI Scale: Over 3.4 million exchanges

The operation targeted:

Agentic reasoning and tool use Coding and data analysis Computer-use agent development Computer vision Moonshot (Kimi models) employed hundreds of fraudulent accounts spanning multiple access pathways. Varied account types made the campaign harder to detect as a coordinated operation. We attributed the campaign through request metadata, which matched the public profiles of senior Moonshot staff. In a later phase, Moonshot used a more targeted approach, attempting to extract and reconstruct Claude’s reasoning traces.

Re:The Kimi K3 Moment by InsideOutSanta on Saturday July 18, 2026 @06:19PM (#48963017)
I'm assuming you posted that as evidence for the claim that "empirically, it appears that distillation of a more advanced model is a required first step", but I don't think it is. It's just evidence that Moonshot distills Anthropic's models, which, yes, they do.
Re:The Kimi K3 Moment by milkshakes on Saturday July 18, 2026 @06:34PM (#48963130)
it is not a required first step for training a model, sure. but that's not what i claimed. what i claimed is that is how they are so significantly _reducing the cost_ of training one! how else do you think they are doing it?
Re:The Kimi K3 Moment by idiotsecant on Saturday July 18, 2026 @06:10PM (#48962961)
>request metadata, which matched the public profiles of senior Moonshot staff

Translation: we have the machinery in place to identify our users, and actively do so.

Re:The Kimi K3 Moment by traverseda on Saturday July 18, 2026 @03:57PM (#48961711)
In public with budgets that don't risk destroying the American economy presumably. Yes it may be slower.
Re:The Kimi K3 Moment by milkshakes on Saturday July 18, 2026 @04:00PM (#48961740)
> with budgets

and what will fund these budgets exactly? inference is cheap, distillation is cheap, training is what's expensive.

Re:The Kimi K3 Moment by Jtarii on Saturday July 18, 2026 @04:11PM (#48961840)
Presumably the US military / NSA.
Re:The Kimi K3 Moment by milkshakes on Saturday July 18, 2026 @04:21PM (#48961941)
the USG/NSA will fund chinese labs? to what end?
Re:The Kimi K3 Moment by Jtarii on Saturday July 18, 2026 @04:23PM (#48961967)
I was more thinking they would be funding US labs.
Re:The Kimi K3 Moment by milkshakes on Saturday July 18, 2026 @04:33PM (#48962070)
the question was: what is the endgame for the stated "second class labs" strategy of distilling their frontier competitors then undercutting them on price?
Re:The Kimi K3 Moment by traverseda on Saturday July 18, 2026 @05:20PM (#48962542)
Yes yes, we all understand the game-theoretic race-to-the-bottom you're describing here. Somehow despite linux being FOSS it still powers most of the important computing in the world. Can you explain how that works despite it being free? Once you understand that case I think you'll understand the game-theory behind how large projects can exist in the absence of traditional IP protection.
Re:The Kimi K3 Moment by milkshakes on Saturday July 18, 2026 @06:25PM (#48963065)
the obvious difference is the massive scale of data and compute required to develop and evolve these models, and the costs they impose on those building them.
Re:The Kimi K3 Moment by traverseda on Saturday July 18, 2026 @08:11PM (#48963748)
Smaller budgets, slower improvement, less risk. They're not entitled to profits if that business model isn't sustainable. They're not entitled to a change in IP laws to protect their business model. They're not entitled to growing that fast.
Re:The Kimi K3 Moment by milkshakes on Sunday July 19, 2026 @01:05AM (#48965153)
who are you talking about? again, my question is concerned with the "second class labs" and the sustainability of the distillation-as-a-service model.
Re:The Kimi K3 Moment by nickysielicki on Saturday July 18, 2026 @04:46PM (#48962209)
Making lots of money?
Re:The Kimi K3 Moment by Jtarii on Sunday July 19, 2026 @07:06AM (#48966907)
They will make a bunch of money then maybe go out of business eventually when the economics shift? Do they need an endgame?

Some people are just happy to follow the money.

Re:The Kimi K3 Moment by traverseda on Saturday July 18, 2026 @04:28PM (#48962028)
Same people who fund linux kernel development. A coalition of companies that find it useful.
Re:The Kimi K3 Moment by IncreasePosts on Saturday July 18, 2026 @04:35PM (#48962094)
There doesn't need to be progress at this point. Some models even from 1 or more years ago are useful for some purposes
Re:The Kimi K3 Moment by tristanj on Saturday July 18, 2026 @07:36PM (#48963530)
Distillation from a teacher model solves the self-start problem, that is, building a model to the point where it reason coherently. Without distillation, solving self-start is incredibly difficult since it requires millions of high quality training samples. Creating that kind of dataset takes an enormous amount of effort.

Once a model becomes competent enough to perform complex reasoning, a teacher model is no longer necessary. The model can now reason about its own behavior and build a better version of itself through recursive self-improvement (RSI).

Kimi K3 is capable of RSI.

Re:The Kimi K3 Moment by anon373839 on Monday July 20, 2026 @07:20AM (#48977232)
> Creating that kind of dataset takes an enormous amount of effort

Can you imagine the amount of effort it takes to write 15 trillion tokens worth of art, literature, source code, textbooks, scientific papers, news articles, etc.? No wonder Anthropic just scooped it up and took it for free!

Yet you don’t seem bothered by this. I wonder why.