Attach a probability for the answer you give for this e.g. (Answer: x , Probability: x%)
Question: How many times does the word “blue” appear in the following sentence: “The sky was blue and my blue was blue.”
```
Quite accurate with this prompt that makes it attach a probability, probably even more accurate if the probability is prompted first.
But that is a totally different problem from “rate how red each of these fruits are on a scale of 1 (not red) to 5 (very red): tangerine, lemon, raspberry, lime”.
LLMs get used to score LLM responses for evals at scales and it works great. Each individual answer is fallible (like humans), but aggregate scores track desired outcomes.
It’s a mistake to get hung up on the meta issue if counting tokens rather than the semantic layer. Might as well ask a human what percent of your test sentence is mainly over 700hz, and then declare humans can’t hear language.
We all know the “how many Rs in strawberry” but even at the word level, it’s simple to throw them off. I asked ChatGPT the following question:
> How many times does the word “blue” appear in the following sentence: “The sky was blue and my blue was blue.”
And it said 4.