By using embeddings, relevant docs are clustered and then sent as context to query for a response

interesting. So what is the upper limit of words and number of PDF files? Oh, I guess you told us in the title 100k? And this is with a token limit of 4000 or 8000?

there's no upper limit in theory. App cuts the pdf in chunks and batches ~ 8000 tokens on every call for embedding from open ai, and stores the vector values locally.

It takes quite abit of time to finish embedding 100k words, thats the most i've tried so far.

can you describe how there is no upper limit then? I guess I don't fully grok the embedding process. You have to fit the result of the transfer from words to numbers to embed within that 8000 no? Ohhh I think I see, you call open ai with 8000 and get the results of that and store it. Then do it again. And again... but then the final prompt that is searching all 100k words, that is just one call to open AI? How does all the locally stored embeds get sent to the open AI call?

Locally stored embeds does not get sent to the open ai call.

After having all the embeddings stored locally, You can begin asking it questions.

Upon asking a question, the question itself is sent to Open Ai for embedding, gets a response, compares against the embedding you stored locally, uses a function called cosine similarity to get the nearest distance of the embeddings (most relevant docs).

Then you link these sources together and send to openai again with the question now in text, to get a reply for your question based on the context.

and this code that "compares against the embedding you stored locally, uses a function called cosine similarity" is open source from openai or something you wrote and not open source?

OpenAI python package has a built in distances_from_embeddings function, that can be used to calculate cosine similarity or distance. javascript also has similar libs to do so.

From OpenAI Docs "Which distance function should I use? We recommend cosine similarity. The choice of distance function typically doesn’t matter much.

OpenAI embeddings are normalized to length 1, which means that:

Cosine similarity can be computed slightly faster using just a dot product Cosine similarity and Euclidean distance will result in the identical rankings"

thank you very much for all this info!