Astra can read and write weights to improve Qwen

Re:Astra can read and write weights to improve Qwen by charm137 on Monday November 02, 2026 @02:18PM (#50342107)

My somewhat uninformed guess is that the weights of the target network (Qwen or Gemma), when they reach the context of the larger editor model (Astra), the attention process is doing some sort of PCA on the weights, conditional on the prompt ("make this better at games"), fetching the relevant weights from its own network (which serves as the larger database), transforming it into whatever latent space the target model is amenable to and pasting them. If this is the right intuition, it is amazing what attention is able to achieve. As a test, I think smaller networks won't be able to make big ones better using the same process.

EDIT: I think the bigger implication is that backprop is not the only game in town.

[ Parent | on HN ]

Re:Astra can read and write weights to improve Qwen by adw on Monday November 02, 2026 @02:31PM (#50342345)

https://xingjianbai.com/reactive-optimizers/

from this month

[ Parent | on HN ]

Re:Astra can read and write weights to improve Qwen by GolDDranks on Monday November 02, 2026 @02:35PM (#50342402)

> fetching the relevant weights from its own network

They don't have usually access to their own weights?

[ Parent | on HN ]

Re:Astra can read and write weights to improve Qwen by theowaway213456 on Monday November 02, 2026 @02:38PM (#50342462)

I think they're saying this is all happening in latent space through pure reasoning, and "fetching" is just a metaphor for that, not a literal tool call. I do believe it's possible for models to learn the capability to "introspect" their own weights during training.

[ Parent | on HN ]

Re:Astra can read and write weights to improve Qwen by charm137 on Monday November 02, 2026 @02:46PM (#50342605)

Yes, I should have been more precise - I meant the information embedded in their weights (which every neural network retrieves to function), not their literal weights.

[ Parent | on HN ]

Re:Astra can read and write weights to improve Qwen by galaxyLogic on Monday November 02, 2026 @02:25PM (#50342245)

So it's like a smaller LLM can learn from a bigger one, but not vice versa.

But shouldn't the model-providers be the ones doing this kind of "post training learning"?

[ Parent | on HN ]

Re:Astra can read and write weights to improve Qwen by charm137 on Monday November 02, 2026 @02:32PM (#50342360)

I don't know if some of the labs already knew of this. But I think there would be a limit to how big a model you can push into the context of the editor models. Typically, with 1M context and say 5120 hidden dimensions supported at fp16 / bf16, you could potentially edit the weights of an fp4 20B param model (roughly).

[ Parent | on HN ]