News for hackers. Stuff that matters.

hashemalsaket

For now, the design is basic:

User to LLM: "Rate this response to the following prompt on a scale of 1-10, where 1 is a poor response and 10 is a great response: [response]"

LLM rates responses of all other LLMs

All other LLMs do the same

Then we take the average score of each response. The LLMs that produced the top 50% of responses will respond again until one response with the highest score remains.

One approach we've been working on is having multiple LLMs score each other. Here is the design with an example of how that works: https://github.com/HashemAlsaket/prompttools/pull/1

In short: Pick top 50% responses, LLMs score each other, repeat until top response remains