A new semantic chunking approach for RAG

Re:A new semantic chunking approach for RAG by refulgentis on Tuesday September 24, 2024 @11:58PM (#41643548)

It'd be interesting to see examples involving the "G" in RAG: I'm wary that having a fine-grained breakdown of, say, 2 vs. 5 sentences, would affect things significantly.

If I'm understanding correctly, you end up with a "turtles all the way down" problem: you need to chunk using some size = K, in order to get the inputs you need to decide bespoke chunk sizes X, Y, and Z.

[ Parent | on HN ]

Re:A new semantic chunking approach for RAG by bbor on Wednesday September 25, 2024 @12:39AM (#41643748)

Yeah but this becomes very, very relevant for larger corpuses. It’s not just about pulling the right sized snippet in the end (really that could be done by an intermediate LLM call…), it’s also about supporting effective vector similarity searches to find the right snippet in the first place.

If you’re embedding all five pages into a single vector, and only two of them are relevant to your query, that makes the search much less effective. AFAIU.

[ Parent | on HN ]

Re:A new semantic chunking approach for RAG by refulgentis on Wednesday September 25, 2024 @12:05PM (#41648916)

Exactly: you have to do all the small embeddings to get the larger embeddings.

Also, in practice, embedding are much smaller than that, the popular MiniLM v2 is 256 words.

Leaving me unsure how this helps

[ Parent | on HN ]