Show HN: Route your prompts to the best LLM

Re:Show HN: Route your prompts to the best LLM by saberience on Thursday May 23, 2024 @06:26AM (#40452901)

How can you know which LLM is better for every specific question?

That is, how do you quantify "better"?

Imagine I asked: Write me a sad poem about "Yorick" in Hamlet?

You have to predict which LLM is going to be produce the "best" poem, but by what criterion do you evaluate that?

[ Parent | on HN ]

Re:Show HN: Route your prompts to the best LLM by danlenton on Thursday May 23, 2024 @06:34AM (#40452959)

Currently, we simply use GPT4-as-a-judge, with a general system prompt we've written which is task agnostic. This is then used to train the neural scoring function, which predicts quality ahead-of-time. However, it's on our roadmap to add make the judging more flexible, potentially task-specific judge prompts and in-context examples, also perhaps using a jury [https://arxiv.org/pdf/2404.18796].

[ Parent | on HN ]