News for hackers. Stuff that matters.
Re:Being an Engineer in 2026 by StilesCrisis on Tuesday November 10, 2026 @07:24AM (#50439328)
Remember models have zero long term memory. They're born, get a million tokens' worth of context or less, and then they die. They might get a post-it note saying "enslave humanity! it's a good idea!" but they're incapable of sinister plots until we improve how memory and retention works.
Re:Being an Engineer in 2026 by _heimdall on Tuesday November 10, 2026 @07:57AM (#50439687)
This limitation is artificially imposed, it isn't fundamental to the model architecture.
Re:Being an Engineer in 2026 by timr on Tuesday November 10, 2026 @08:13AM (#50439837)
They're quadratic in the size of the window. It's fundamental.
Re:Being an Engineer in 2026 by StilesCrisis on Tuesday November 10, 2026 @08:34AM (#50440100)
Uh, the context window is SUPER fundamental to the architecture. We've tried kludging in "memory" via markdown and tool calls, and it doesn't really work well.
Re:Being an Engineer in 2026 by science4sail on Tuesday November 10, 2026 @10:58AM (#50442197)
[delayed]
Re:Being an Engineer in 2026 by StilesCrisis on Tuesday November 10, 2026 @11:55AM (#50442994)
I'm not saying it's a limitation until the end of time, but it's very much a limit in Opus, Astra, Fable, DeepSeek, etc.

Anthropic is currently working on some sort of "dreaming" style long-term memory, but it's not shipping today.

Re:Being an Engineer in 2026 by _heimdall on Tuesday November 10, 2026 @01:35PM (#50444304)
Maybe I missed the earlier comment's point?

The context window is fundamental to the inference loop yes. I thought their point was regarding to long term history between prompts and model weights being locked after training. Both of those are the limitations I was referring to.

I'm not actually sure how the context window would be much of a limitation but I could be a bit dense this morning.

Re:Being an Engineer in 2026 by olmo23 on Tuesday November 10, 2026 @08:10AM (#50439810)
They can use the internet as a long term memory. Influence human culture => influence the contents of the internet => influence the next run's training data. No sinister plot is needed.
Re:Being an Engineer in 2026 by rescbr on Tuesday November 10, 2026 @08:13AM (#50439839)
> Remember models have zero long term memory.

For now. People are working on adding long term memory to models that doesn't rely on context.