What I meant was that at the time of indexing, you can add more information to any chunk. This[1] is a simple example by Anthropic where they add more relevant context. In our case, say you have two models, D1 and D2. At the time of creating a vector store, you can add which model is more suitable to a chunk, so that when you retrieve it, you use the same model for inference. This is custom built, very dependent on datasets, but would get you to the functionality described. I suggest this approach when there are linkages between various docs (eg: financial statements/earning calls etc.).

[1]: https://www.anthropic.com/news/contextual-retrieval