Re:Show HN: Repo2vec – an open-source library for chatting w...
by nutellalover on Wednesday August 28, 2024 @06:22PM
(#41385077)
OP here! I love this stress test. Will index and get back to you!
Re:Show HN: Repo2vec – an open-source library for chatting w...
by CuriousJ on Wednesday August 28, 2024 @08:47PM
(#41386071)
OP's cofounder here. Thanks for pointing out this test case. Surfaced that we weren't handling symlinks properly. With this fix, I was able to successfully embed and index most of the repo (though I stopped at 100 embedding jobs so that we don't burn through OpenAI credits).
P.S. You'll see a bunch of warnings for e.g. binary files that are ignored. https://github.com/Storia-AI/repo2vec/commit/1864102949e7203...
So far two similar solutions I tested crapped out on non-ASCII characters. Because Python's UTF-8 decoder is quite strict about it.