hashemalsaket
Re:Show HN: PromptTools – open-source tools for evaluating LLMs and ve...
on Tuesday August 01, 2023 @01:50PM (#36959640)
For now, the design is basic:
User to LLM: "Rate this response to the following prompt on a scale of 1-10, where 1 is a poor response and 10 is a great response: [response]"
LLM rates responses of all other LLMs
All other LLMs do the same
Then we take the average score of each response. The LLMs that produced the top 50% of responses will respond again until one response with the highest score remains.
Re:Show HN: PromptTools – open-source tools for evaluating LLMs and ve...
on Tuesday August 01, 2023 @01:28PM (#36959291)
One approach we've been working on is having multiple LLMs score each other. Here is the design with an example of how that works: https://github.com/HashemAlsaket/prompttools/pull/1
In short: Pick top 50% responses, LLMs score each other, repeat until top response remains