can you describe how there is no upper limit then? I guess I don't fully grok the embedding process. You have to fit the result of the transfer from words to numbers to embed within that 8000 no? Ohhh I think I see, you call open ai with 8000 and get the results of that and store it. Then do it again. And again... but then the final prompt that is searching all 100k words, that is just one call to open AI? How does all the locally stored embeds get sent to the open AI call?

Locally stored embeds does not get sent to the open ai call.

After having all the embeddings stored locally, You can begin asking it questions.

Upon asking a question, the question itself is sent to Open Ai for embedding, gets a response, compares against the embedding you stored locally, uses a function called cosine similarity to get the nearest distance of the embeddings (most relevant docs).

Then you link these sources together and send to openai again with the question now in text, to get a reply for your question based on the context.

and this code that "compares against the embedding you stored locally, uses a function called cosine similarity" is open source from openai or something you wrote and not open source?

OpenAI python package has a built in distances_from_embeddings function, that can be used to calculate cosine similarity or distance. javascript also has similar libs to do so.

From OpenAI Docs "Which distance function should I use? We recommend cosine similarity. The choice of distance function typically doesn’t matter much.

OpenAI embeddings are normalized to length 1, which means that:

Cosine similarity can be computed slightly faster using just a dot product Cosine similarity and Euclidean distance will result in the identical rankings"

thank you very much for all this info!