News for hackers. Stuff that matters.
Re:You probably shouldn't use OpenAI's embeddings by mustacheemperor on Thursday March 30, 2023 @05:03PM (#35378792)
Could anyone point me towards a relatively beginner-friendly guide to do something like

>download all my tweets (about 20k) and build a semantic searcher on top ?

How can utilize 3rd party embeddings with OpenAI's LLM API? Am I correct to understand from this article that this is possible?

Re:You probably shouldn't use OpenAI's embeddings by diego on Thursday March 30, 2023 @05:07PM (#35378831)
That's exactly what I did here. https://github.com/dbasch/semantic-search-tweets
Re:You probably shouldn't use OpenAI's embeddings by mustacheemperor on Thursday March 30, 2023 @05:16PM (#35378941)
Thank you! Comparing this and the link the other commenter posted, what handles the actual search querying? Does instructor-xl include an LLM in addition to the embeddings? The other commenter's repo uses Pinecone for the embeddings and OpenAI for the LLM.

My apologies if I am completely mangling the vocabulary here - I have an, at best, rudimentary understanding of this stuff that I am trying to hack my education on.

Edit: If you're at the SF meetup tomorrow, I'd happily buy you a beverage in return for this explanation :)

Re:You probably shouldn't use OpenAI's embeddings by eternalban on Thursday March 30, 2023 @06:28PM (#35379687)
It's in the repo:

You first create embeddings. What is this? It's an n-dimensional vector space with your tweets 'embedded' in that space. Each word is an n-dimensional vector in this space. The vectorization is supposed to maintain 'semantic distance'. Basically, if two words are very close in meaning or related (by say frequently appearing next to each other in corpus) they should be 'close' in some of those n-dimensions as well. The result at the end is the '.bin' file, the 'semantic model' of your corpus.

https://github.com/dbasch/semantic-search-tweets/blob/main/e...

For semantic search, you run the same embedding algorithm against the query and take the resultant vectors and do similarity search via matrix ops, resulting in a set of results, with probabilities. These point back to the original source, here the tweets, and you just print the tweet(s) that you select from that result set (here the top 10).

https://github.com/dbasch/semantic-search-tweets/blob/main/s...

Experts can chime in here but there are knobs such as 'batch size' and the functions you use to index. (cosine was used here.)

So the various performance dimensions of the process should also be clear. There is a fixed cost of making the embeddings of your data. There is a per-op embedding of your query, and then running the similarity algorithm to find the result set.

Re:You probably shouldn't use OpenAI's embeddings by mustacheemperor on Thursday March 30, 2023 @06:51PM (#35379950)
Thank you for this walkthrough, and for citing the code alongside!
Re:You probably shouldn't use OpenAI's embeddings by eternalban on Thursday March 30, 2023 @07:09PM (#35380116)
hth
Re:You probably shouldn't use OpenAI's embeddings by celestialcheese on Thursday March 30, 2023 @05:09PM (#35378854)
langchain and llama-index are two big opensource projects which are great for buildign this type of thing.

https://github.com/mayooear/gpt4-pdf-chatbot-langchain for example

Re:You probably shouldn't use OpenAI's embeddings by mustacheemperor on Thursday March 30, 2023 @05:17PM (#35378942)
Cheers, thank you!
Re:You probably shouldn't use OpenAI's embeddings by sroussey on Thursday March 30, 2023 @06:09PM (#35379466)
I have a system that download all my data from Google, Facebook, Twitter, and others. Geo data is fun to look at, but now the text and images have some more meaning to gleam. I’m thinking about going back to it. Not sure how to package a bunch of python stuff in an app though.
Re:You probably shouldn't use OpenAI's embeddings by devxpy on Thursday March 30, 2023 @06:28PM (#35379686)
Re:You probably shouldn't use OpenAI's embeddings by jeadie on Thursday March 30, 2023 @07:39PM (#35380386)