Smart:

> However, using an exact match approach for LLM caches is less effective due to the complexity and variability of LLM queries, resulting in a low cache hit rate. To address this issue, GPTCache adopt alternative strategies like semantic caching. Semantic caching identifies and stores similar or related queries, thereby increasing cache hit probability and enhancing overall caching efficiency.

> GPTCache employs embedding algorithms to convert queries into embeddings and uses a vector store for similarity search on these embeddings. This process allows GPTCache to identify and retrieve similar or related queries from the cache storage, as illustrated in the Modules section.

While I wont pretend to fully understand this I assume it means more cache hits for similar but not exactly the same queries?

You are right, most LLM queries won't be exact match, so you will need semantic search for hit those similar but not same questions

You use a vector database for similarity search right, which vector DB do you use, do you your product ziliz?

The current implementations use Milvus and FAISS, but we have opens for other open-source vector databases as well.

Nice idea, thanks for publishing this, wondered for a long time why this wasn't yet in something like langchain (put it down to the storage aspect, despite query/query_with_sources being released recently). Having a cache system like this for the last 2 years we ran GPT-3 before the hype came has prevented us switching to langchain.