News for hackers. Stuff that matters.

A comment on Language Models Need Sleep. Parent comment

Re:Language Models Need Sleep by colechristensen on Tuesday May 26, 2026 @12:34PM (#48282065)
No, they're actually training weights based on context before compaction. Context is context, this is splitting the model into persistent weights and malleable ones which are periodically updated.
[ Parent | on HN ]
Re:Language Models Need Sleep by delis-thumbs-7e on Tuesday May 26, 2026 @12:39PM (#48282151)
Wouldn’t that be extremely computationaly expensive considering how resource incentive training is?
[ Parent | on HN ]
Re:Language Models Need Sleep by colechristensen on Tuesday May 26, 2026 @12:42PM (#48282190)
No, training a state of the art model involves training on the order of 10 trillion tokens.

We're talking about a step that updates weights based on say between 10k and 1M tokens.

[ Parent | on HN ]
Re:Language Models Need Sleep by delis-thumbs-7e on Tuesday May 26, 2026 @12:44PM (#48282222)
I learned something. Thank you!
[ Parent | on HN ]