News for hackers. Stuff that matters.

A comment on Show HN: Chonkie – A Fast, Lightweight Text Chunking Library for RAG. Parent comment

Re:Show HN: Chonkie – A Fast, Lightweight Text Chunking Libr... by nostrebored on Sunday November 10, 2024 @12:42PM (#42101409)
Poorly, just like it does for text.

Chunking is easily where all of these problems die beyond PoC scale.

I’ve talked to multiple code generation companies in the past week — most are stuck with BM25 and taking in whole files.

[ Parent | on HN ]
Re:Show HN: Chonkie – A Fast, Lightweight Text Chunking Libr... by potatoman22 on Sunday November 10, 2024 @09:09PM (#42104062)
What do they use BM25 for? RAG?
[ Parent | on HN ]
Re:Show HN: Chonkie – A Fast, Lightweight Text Chunking Libr... by nostrebored on Tuesday November 12, 2024 @02:28AM (#42113505)
Correct -- finding the correct functions and files to include
[ Parent | on HN ]