News for hackers. Stuff that matters.

A comment on Building more with GPT-5.1-Codex-Max. Parent comment

Re:Building more with GPT-5.1-Codex-Max by int_19h on Thursday November 20, 2025 @12:50AM (#45989387)
> As the context fills up, the LLM places less attention on further and further back in the context, that's why the LLM seems dumber and dumber as a conversation goes on.

This is not entirely true. They pay the most attention to the things that are the earliest in history and the most recent in it, while the middle between the two is where the dip is. Which basically means that the system prompt (which is always on top) is always going to have attention. Or, perhaps, it would be more accurate to say that because they are trained to follow the system prompt - which comes first - that's what they do.

[ Parent | on HN ]
Re:Building more with GPT-5.1-Codex-Max by boredtofears on Thursday November 20, 2025 @11:29AM (#45994420)
Do you have any idea why they (seemingly randomly) will drop the ball on some system prompt instructions in longer sessions?
[ Parent | on HN ]
Re:Building more with GPT-5.1-Codex-Max by int_19h on Friday November 21, 2025 @01:55PM (#46007617)
Larger contexts are inherently more attention-taxing, so the more you throw at it, the higher the probability that any particular thing is going to get ignored. But that probability still varies from lower at the beginning to higher in the middle and back to lower in the end.
[ Parent | on HN ]