News for hackers. Stuff that matters.
Re:Embeddings: What they are and why they matter by simonw on Tuesday October 24, 2023 @11:52PM (#38008948)
OpenAI's embedding model works up to 8,000 tokens. The sentence-transformers ones are mostly smaller than that, though there's a new model that just came out that can handle 8,000: https://huggingface.co/jinaai/jina-embeddings-v2-base-en

People tend to "chunk" larger documents - the chunking strategy that's best is very dependent on what you are using the embeddings for. I've found it frustratingly hard to find really good guidance as to chunking strategies.

I've had good results for Q&A chunking my blog content up into paragraph sized chunks, as described here: https://simonwillison.net/2023/Oct/23/embeddings/#answering-... - but I'm not ready to say that's a universally good practice.

Re:Embeddings: What they are and why they matter by CityOfThrowaway on Wednesday October 25, 2023 @12:37AM (#38009179)
I'm also curious about how to think about chunking for Doc-to-Doc similarity.

Say I have an input document that I want to use as my search query and a database of potential results documents.

It's not clear to me that I should simply embed both the input document and each of the results documents. If the documents contain a variety of ideas, I'd be nervous they wind getting embedded into some generic space.

But I don't have any good ideas as to what type of chunk-match-and-aggregate strategy might work well here.

Would love ideas from folks that have done stuff like this!

Re:Embeddings: What they are and why they matter by Swizec on Wednesday October 25, 2023 @01:06AM (#38009340)
> If the documents contain a variety of ideas

I've found chunking by sub-headings works really well.

But here's a dirty secret I've learned as a writer: Your document can only ever contain 1 idea. That's the most that human readers can manage.

Re:Embeddings: What they are and why they matter by llmllmllm on Wednesday October 25, 2023 @12:53AM (#38009263)
I have found success with chunking with 100 tokens, preceeded by the last 10 tokens of the previous chunk, and the first 10 tokens of the next chunk, 120 tokens total. I generate an embedding for each, then compare that to embedding(s) derived from the input query.

How to generate embeddings from the input query well is where one's focus should be IMO. An example: "don't mention x" being turned into filtering out / de-emphasizing chunks that align with the embedding for x.

I've been using these techniques along with pgvector and OpenAI's embeddings for https://flowch.ai and it works really well. A user uploads a document or uses the Chrome Extension on a webpage and FlowChai chunks up the content, generates embeddings, builds up a RAG context and then produces a report based on the user's prompt.

I hope that helps show a real world example. You're welcome to play with FlowChai for free to see how it works in practice at the application level.

Re:Embeddings: What they are and why they matter by tayo42 on Wednesday October 25, 2023 @10:59AM (#38013541)
> I've found it frustratingly hard to find really good guidance as to chunking strategies.

Atleast I'm not alone. Paragraph can make sense, I was worried about paragraphs that are related in concepts, they're not always standalone section of thoughts in real writing.

Maybe if two paragraphs meet some similarity threshold you can merge them.