News for hackers. Stuff that matters.

A comment on Show HN: Chonkie – A Fast, Lightweight Text Chunking Library for RAG.

Re:Show HN: Chonkie – A Fast, Lightweight Text Chunking Libr... by petesergeant on Sunday November 10, 2024 @01:29PM (#42101684)
> What other chunking strategies would be useful for RAG applications?

I’m using o1-preview for chunking, creating summary subdocuments.

[ Parent | on HN ]
Re:Show HN: Chonkie – A Fast, Lightweight Text Chunking Libr... by bhavnicksm on Sunday November 10, 2024 @01:36PM (#42101719)
That's pretty cool! I believe a research paper called LumberChunker recently evaluated that to be pretty decent as well.

Thanks for responding, I'll try to make it easier to use something like that in Chonkie in the future!

[ Parent | on HN ]
Re:Show HN: Chonkie – A Fast, Lightweight Text Chunking Libr... by petesergeant on Sunday November 10, 2024 @02:40PM (#42102137)
Ah, that's an interesting paper, and a slightly different approach to what I'm doing, but possibly a superior one. Thanks!
[ Parent | on HN ]