News for hackers. Stuff that matters.
Re:voyage-code-3 by potatoman22 on Monday January 13, 2025 @08:49PM (#42692459)
What's their ranking on SWEBench?
Re:voyage-code-3 by swyx on Monday January 13, 2025 @09:08PM (#42692599)
for an embedding model, this is a very very bad question, to the point where i am wondering if you are trolling and i missed the joke

i can be constructive but i dont have space in the margins of this paper

Re:voyage-code-3 by moojacob on Monday January 13, 2025 @09:14PM (#42692646)
Pardon my ignorance, but are you saying that embedding model leaderboards aren't useful? When I look at https://huggingface.co/spaces/mteb/leaderboard I see voyage's embedding model is at the top. Does that mean they have the best model?
Re:voyage-code-3 by fzliu on Monday January 13, 2025 @09:18PM (#42692676)
Although voyage-3-m-exp is at the top of the leaderboard, I would not use it for production use cases unless your data is extremely similar to "in-domain" MTEB data. Check out this post from Nils (one of the authors of the original MTEB paper) for more info: https://x.com/Nils_Reimers/status/1870812625505849849

voyage-3-large will work better for almost all real-world production use cases.

Re:voyage-code-3 by moojacob on Wednesday January 15, 2025 @03:14PM (#42716297)
Which model would you choose?
Re:voyage-code-3 by evilduck on Monday January 13, 2025 @09:21PM (#42692698)
They did not claim that leaderboards were useless. There’s a big difference between an embedding model leaderboard and the SWEbench leaderboard though, given that they measure completely different categories of models.
Re:voyage-code-3 by potatoman22 on Tuesday January 14, 2025 @12:16PM (#42700312)
Woops. I should have done more reading before posting.
Re:voyage-code-3 by fzliu on Monday January 13, 2025 @09:13PM (#42692641)
We have a repurposed version of SWE-bench Lite for retrieval; you can see the numbers in the last column of this spreadsheet: https://docs.google.com/spreadsheets/d/1Q5GDXOXueHuBT9demPrL...