Yeah, what is being stored that is available between inferences?

Nothing, if by "inference" you mean single API call. The underlying algorithm starts (very roughly) by processing the prompt into a set of vectors, which are then used repeatedly to generate tokens one after the other. What's meant here is that the act of generating and using those prompt vectors has an effect sort of like regular training. It doesn't mean anything is being written to disk afterwards, though of course you could write those vectors to disk and re-use them. It just doesn't make much sense at the moment. Doing that isn't enough to give you infinite context windows.

An hypothesis is that in the latent space there is something like a branching factor that can be used in context learning to select the main tree for others layers. So a LLM is able to have the knowledge of many specialized smallers LLM and select the appropriate one by way of giving values to the branching factor. The branching factor could be some combination of attention layers operating in the latent space of previous layers.

Remember the ancient “king - man = queen” word vectors? This is just that but produced by a model rather than manually adding vectors and finding the nearest one.

Depending on what one understands by “new passes through the model” it may not be.