Show HN: Repo2vec – an open-source library for chatting with any codebase

Re:Show HN: Repo2vec – an open-source library for chatting w... by siamese_puff on Wednesday August 28, 2024 @09:53PM (#41386462)

Not an expert, but OP is right and this is generally a known issue with large windows and RAG. Small chunks are usually best. Also how you chunk is important. OP - what’s the most optimal way to parse/chunk code snippets?

[ Parent | on HN ]

Re:Show HN: Repo2vec – an open-source library for chatting w... by jshobrook on Wednesday August 28, 2024 @10:07PM (#41386531)

You can use the AST to chunk the code: https://docs.sweep.dev/blogs/chunking-2m-files

[ Parent | on HN ]

Re:Show HN: Repo2vec – an open-source library for chatting w... by CuriousJ on Wednesday August 28, 2024 @10:09PM (#41386547)

We're using an improvement over this exact blogpost actually. We started from there, but weren't happy that some of the chunks were really small (and they would undeservedly get surfaced to the top). So we added some extra logic to merge the siblings if they're small.

https://github.com/Storia-AI/repo2vec/blob/1864102949e720320...

[ Parent | on HN ]