The analogy I can think of here is a pointer, but AFAIK the context would always need to go along with the prompt unless you could tweak internal state to bias towards the context.

Otherwise, it might make sense to have a separate routine which compresses the context as efficiently as possible. Auto encoder?