Re:Show HN: Danswer – Open-source question answering across ...
by darkteflon on Monday July 10, 2023 @07:34PM
(#36674522)
Chunking is very important but might, I feel, best be contextualised as one aspect of the bigger substantive challenge, which is how to prevent false negatives at the context retrieval stage - a.k.a. how to ensure your (vector? hybrid?) search returns all relevant context to the LLM’s context window.
Would you mind saying a few words on how Danswer approaches this?
Additionally, the indexing process is setup as a composable pipeline under the hood. It would be fairly trivial to plug in different chunkers for different sources as needed in the future.