Re:Show HN: Chat with multiple (100k words) PDFs privately
by andrewfromx on Tuesday April 11, 2023 @01:22PM
(#35528201)
and this code that "compares against the embedding you stored locally, uses a function called cosine similarity" is open source from openai or something you wrote and not open source?
Re:Show HN: Chat with multiple (100k words) PDFs privately
by simtj on Tuesday April 11, 2023 @01:41PM
(#35528473)
OpenAI python package has a built in distances_from_embeddings function, that can be used to calculate cosine similarity or distance. javascript also has similar libs to do so.
From OpenAI Docs "Which distance function should I use? We recommend cosine similarity. The choice of distance function typically doesn’t matter much.
OpenAI embeddings are normalized to length 1, which means that:
Cosine similarity can be computed slightly faster using just a dot product Cosine similarity and Euclidean distance will result in the identical rankings"
Re:Show HN: Chat with multiple (100k words) PDFs privately
by andrewfromx on Tuesday April 11, 2023 @01:49PM
(#35528587)
thank you very much for all this info!
After having all the embeddings stored locally, You can begin asking it questions.
Upon asking a question, the question itself is sent to Open Ai for embedding, gets a response, compares against the embedding you stored locally, uses a function called cosine similarity to get the nearest distance of the embeddings (most relevant docs).
Then you link these sources together and send to openai again with the question now in text, to get a reply for your question based on the context.