Re:Changes in the system prompt between Claude Opus 4.6 and 4.7
by pests on Sunday April 19, 2026 @10:12PM
(#47829608)
The state of the system can be cached after the system prompt is calculated and all new chats start from that state. O(n^2) is not great but apparently its fine at these context lengths and I'm sure this is a factor in their minimum prompt cost. Advances like grouped query or multi head attention or sparse attention will eventually get rid of that exponential, hopefully.
[ Parent
| on HN ]
Re:Changes in the system prompt between Claude Opus 4.6 and 4.7
by sigmoid10 on Monday April 20, 2026 @03:25AM
(#47831305)
That's not how it works. The system prompt doesn't "get calculated first" or anything. You combine it with the user prompt and then run the generation for the first new token on that thing, which basically boils down to one huge matmul that runs in parallel. So you can literally just cache a part of the input matrices for the first step and then you'll very quickly run into n^2 complexity.
Re:Changes in the system prompt between Claude Opus 4.6 and 4.7
by pests on Tuesday April 21, 2026 @10:49AM
(#47849601)
The system prompt will always match in the prefix cache. I just meant it could be prefilled before any user queries on completely different hardware. Then you are only dealing with the n^2 only for the actual user prompt. We're in agreeance I think.