Re:Astra can read and write weights to improve Qwen
by triceratops on Monday November 02, 2026 @04:37PM
(#50344168)
In my opinion that's very handwavey.
Re:Astra can read and write weights to improve Qwen
by tomrod on Monday November 02, 2026 @06:02PM
(#50345321)
Definitely see where you might feel so. I provided an analogy that may help anchor - why ML fails extrapolation is left as an exercise to the determined reader! :)
The surface for "interpolation" is fairly broad for LLMs but it isn't infinite -- extrapolation still exists!
Re:Astra can read and write weights to improve Qwen
by 8note on Monday November 02, 2026 @09:57PM
(#50347458)
the most obvious example was counting the Rs in strawberry.
figuring out what is and isnt in distribution is quite difficult though given that theres so much data passed in, and transformers have some amount of combining it together
theres some really obvious things though, like picking nonsense tokens together, or things outside of the tokenspace entirely
you could also construct a string that is a series of the least likely tokens in a row. you wont get it to output the next token, despite it being quite clear how you did it
I think it can patch together components of tasks represented in its training data to do demonstrably new things, yes, which does count as "perform tasks not in its training data" with the limitation that some form of the task must be present. Example: writing an ode to a sales lead for a job well done in 2032 for the first quarter. Odes exist, the concept of sales performance exist, good job performance, and notions of time. Stringing those together are good examples of interpolation but not extrapolation, as it were.