The 'Astra can read and write weights to improve Qwen' post is misleading

Re:The 'Astra can read and write weights to improve Qwen' po... by himata4113 on Tuesday November 03, 2026 @09:02AM (#50352103)

It doesn't really matter how accurate the post is, what it shows is what one could assume how you'd end up with a system such as RSI, how you could prune harmful capabilities from models and build a more efficient training system with biases including and not limited to emergent behavior in astra that might have come from something they've never even trained the model on regarless if ML engineers can already do similar things.

None of the previous studies applies because Astra simply did not exist and this is worthwhile NEW knowledge that has been publicized and it is very interesting to see how it can be compared with other models as both openai and anthropic are keeping any and all RSI development completely private. The work is groundbreaking in a way that there will be a lot more experiments of this nature which will contribute to increasing the public knowledge of such systems.

[ Parent | on HN ]

Re:The 'Astra can read and write weights to improve Qwen' po... by skew-aberration on Tuesday November 03, 2026 @09:08AM (#50352179)

How do you prune out the 'fails to prune out behaviors' behavior? If my model is failing to prune it out, can another model prune out that behavior?

[ Parent | on HN ]

Re:The 'Astra can read and write weights to improve Qwen' po... by himata4113 on Tuesday November 03, 2026 @09:11AM (#50352222)

With normal training you're simply training for higher loss for things you don't want while keeping the loss low for things you do. Same would apply here except you would target low eval for behavior you do not want the model to have rather than trying to increase eval.

[ Parent | on HN ]

Re:The 'Astra can read and write weights to improve Qwen' po... by giancarlostoro on Tuesday November 03, 2026 @09:15AM (#50352272)

> how you could prune harmful capabilities from models and build a more efficient training system

In theory, if OpenAI or Anthropic wanted to release a new open weight model, they could scrutinize it to remove the most horrendous things from the model.

[ Parent | on HN ]

Re:The 'Astra can read and write weights to improve Qwen' po... by duplessitous on Tuesday November 03, 2026 @09:17AM (#50352293)

If it isn't accurate, it isn't showing anything. The Qwen math claim is entirely false in its presentation, it unblocked an existing capability. It did not instill a new one which is what an RSI loop would actually have to do

"[...] how you could prune harmful capabilities from models [...]"

We already do this, abliteration is the term that describes the pruning of capabilities or safeguards for models

"[...] not limited to emergent behavior in astra that might have come from something they've never even trained the model [...]"

If I was attempting to build RSI, I would have these exact types of problems in my RL datasets. There is no proof this is emergent behavior.

[ Parent | on HN ]