News for hackers. Stuff that matters.

A comment on Show HN: Chonkie – A Fast, Lightweight Text Chunking Library for RAG.

Re:Show HN: Chonkie – A Fast, Lightweight Text Chunking Libr... by samlinnfer on Sunday November 10, 2024 @11:55AM (#42101198)
How does it work for code? (Chunking code that is)
[ Parent | on HN ]
Re:Show HN: Chonkie – A Fast, Lightweight Text Chunking Libr... by nostrebored on Sunday November 10, 2024 @12:42PM (#42101409)
Poorly, just like it does for text.

Chunking is easily where all of these problems die beyond PoC scale.

I’ve talked to multiple code generation companies in the past week — most are stuck with BM25 and taking in whole files.

[ Parent | on HN ]
Re:Show HN: Chonkie – A Fast, Lightweight Text Chunking Libr... by potatoman22 on Sunday November 10, 2024 @09:09PM (#42104062)
What do they use BM25 for? RAG?
[ Parent | on HN ]
Re:Show HN: Chonkie – A Fast, Lightweight Text Chunking Libr... by nostrebored on Tuesday November 12, 2024 @02:28AM (#42113505)
Correct -- finding the correct functions and files to include
[ Parent | on HN ]
Re:Show HN: Chonkie – A Fast, Lightweight Text Chunking Libr... by bhavnicksm on Sunday November 10, 2024 @01:27PM (#42101672)
Right now, we haven't worked on adding support for code -- some things like comments (#, //) have punctuations that adversely affect chunking, along with indentation and other issues.

But, it's on the roadmap, so please hold on!

[ Parent | on HN ]