CuriousJ

Show HN: Code retrieval findings from a real-world benchmark

Posted by CuriousJ on Monday September 23, 2024 @01:11PM
from the show-hn dept.

https://github.com/Storia-AI/sage/tree/main/benchmarks/retrieval

While building our OSS chat-with-your-codebase system, we faced so many choices: chunking strategies, embeddings, retrieval algorithms, rerankers, etc.Since we don't want to make decisions based on vibes, and because academic benchmarks are somewhat contrived, we made our own. Our dataset consists of 1,000 questions about Hugging Face's Transformers library, where each question requires 1-3...

( Read More... | no comments | 2 points | on HN )

Re:Show HN: Repo2vec – an open-source library for chatting with any co... on Wednesday August 28, 2024 @10:18PM (#41386600)

We noticed an interesting phenomenon related to the size of the repo. The bigger it is, the more its utility skews towards learning how to use the library as opposed to how to change it, i.e. for the big repos the chat is more useful for users than developers/maintainers.

Re:Show HN: Repo2vec – an open-source library for chatting with any co... on Wednesday August 28, 2024 @10:14PM (#41386576)

Retrieval is rarely ever evaluated in isolation. Academics would indirectly evaluate it by how much it improved question answering. The really cool thing at Google is that there were so many products and use cases (beyond the academic QA benchmarks) that would indirectly tell you if retrieval is useful. Much harder to do for smaller companies with a smaller suite of products and user bases.

Re:Show HN: Repo2vec – an open-source library for chatting with any co... on Wednesday August 28, 2024 @10:09PM (#41386547)

We're using an improvement over this exact blogpost actually. We started from there, but weren't happy that some of the chunks were really small (and they would undeservedly get surfaced to the top). So we added some extra logic to merge the siblings if they're small.

https://github.com/Storia-AI/repo2vec/blob/1864102949e720320...

Re:Show HN: Repo2vec – an open-source library for chatting with any co... on Wednesday August 28, 2024 @10:07PM (#41386533)

This paper shows that 200-800 is the ideal chunk size; if you go above, the model starts getting confused / distracted. https://arxiv.org/pdf/2406.14497

Re:Show HN: Repo2vec – an open-source library for chatting with any co... on Wednesday August 28, 2024 @09:05PM (#41386191)

OP's cofounder here. The nice thing is that a lot of repos include the documentation as well, so it comes for free by simply indexing the repo (like huggingface/transformers for instance).

Re:Show HN: Repo2vec – an open-source library for chatting with any co... on Wednesday August 28, 2024 @09:01PM (#41386167)

OP's cofounder here. Yes, we started with what we perceived as highest quality (OpenAI embeddings + Claude autocompletions), but will definitely make our way to local/OSS. The code is super modular so hopefully the community will help as well.

Re:Show HN: Repo2vec – an open-source library for chatting with any co... on Wednesday August 28, 2024 @08:57PM (#41386149)

OP's cofounder here. For us, OpenAI embeddings worked best. When building a system that has many points of failure, I like to start with the highest quality ones (even if they're expensive / lack privacy) just to get an upper threshold of how good the system can be. Then start replacing pieces one by one and measure how much I'm losing in quality.

P.S. I worked on BERT at Google and have PTSD from how much we tried to make it work for retrieval, and it never really did well. Don't have much experience with BGE though.

Re:Show HN: Repo2vec – an open-source library for chatting with any co... on Wednesday August 28, 2024 @08:50PM (#41386100)

Depending on how large your codebase is, that could get pricey, at least for now. But it's probably just a matter of time until it all gets dirt cheap.

Re:Show HN: Repo2vec – an open-source library for chatting with any co... on Wednesday August 28, 2024 @08:47PM (#41386071)

OP's cofounder here. Thanks for pointing out this test case. Surfaced that we weren't handling symlinks properly. With this fix, I was able to successfully embed and index most of the repo (though I stopped at 100 embedding jobs so that we don't burn through OpenAI credits).

P.S. You'll see a bunch of warnings for e.g. binary files that are ignored. https://github.com/Storia-AI/repo2vec/commit/1864102949e7203...