If tokens 1:3000 are the same you are going to be doing the same work processing them over and over; and then changing the results to adapt to the user tokens at the end.
Every time a token is processed in a transformer, the model computes its attention relative to ALL other tokens in the sequence. This is a key difference from sequential, step-by-step processing where previous states are incrementally built upon (so your :n, :n-1 etc). The attention mechanism in transformers recalculates the relationships for EACH token with ALL other tokens EVERY time any part of the input sequence changes.
When new tokens are added to the sequence (for example user input after a system prompt), the attention relationships for ALL PREVIOUS tokens can change. This is because the context provided by the new tokens can alter the relevance and interpretation of the earlier tokens. As such, the attention scores and subsequently the output embeddings for all tokens are recalculated to integrate this new information. So as you can see, you are not doing the same work over and over.
I hope this helps you better understand how transformers work.
You want to reuse the embedding representation of the prefix prompt before the new user tokens are confiscated, otherwise you are recalculating the embedding over and over and over. God forbid it’s not a small prefix but a huge document.