You can use semantic search, then feed that into the LLM.

There are many solutions already, look into Haystack by deepset, or if you are up for a challenge, you could make something in Langchain.