t5-notdiamond

In offline training of our router, we run extensive cross-domain evaluations to determine when a smaller model can handle a request without any quality loss relative to more powerful models. In an online setting like our chat app, there's probably some more rigorous post-hoc analysis we could do on response quality—could make for a good follow-up post.

Hey hn, this is Tomás, founder of Not Diamond. Today we're open-sourcing RoRF (Routing on Random Forests), a pairwise AI model router that beats all closed and open-source approaches, along with 12 pre-trained model routers:

Hugging Face: https://lnkd.in/gE4FKZWR Github: https://lnkd.in/gn2uKWaY Blog: https://lnkd.in/gutaamm4

When routing between two strong models, RoRF achieves higher accuracy than either individual model at significant cost reductions. When routing between a strong and a weak model pair, we outperform other routing approaches at a lower cost.

The architecture for RoRF is based on a Random Forest Classifier and supports both open-source embedding models like Jina which you can run locally, or closed-source embedding models like Voyage or OpenAI for lower compute requirements.

We developed RoRF earlier this year as a previous iteration of the more powerful router we released through our API. Excited to share this :)

Thanks Sami :) Has been awesome to be able to share our progress with you

Glad to hear that

Awesome, lmk what you think

Thank you—really glad we could work together on this

Not yet—very much on the roadmap though

Hey, thanks for checking it out! You can find some more info on how Not Diamond works here: https://notdiamond.readme.io/docs/how-not-diamond-works

Hey HN, glad to see this here—I’m the founder and CEO of Not Diamond. Not Diamond makes it super easy to train your own custom AI model routers on your data to outperform any single model by intelligently routing to the highest-quality model for each query. We beat every foundation model on every major benchmark at a lower cost and latency.

To train a router, you literally just upload a dataset with your inputs and eval scores for different models. It’s completely agnostic to your choice of scoring metrics, frameworks, or tools. And if you don’t have your own eval data, you can still use Not Diamond’s base router out of the box—it takes <5m to set up.

Some other features worth noting:

• Python, TypeScript, and REST API support

• Option to route to faster/cheaper models when doing so doesn’t impact quality

• Joint prompt optimization interface

• Online, real-time personalization to hyperpersonalize model recommendations to individual end users

• Blazing fast inference speeds (<100ms)

• Easy deployments to your private infra

Would love to hear what folks think.