News for hackers. Stuff that matters.
Re:Gemini 2.5 Flash by omneity on Thursday April 17, 2025 @04:48PM (#43722075)
In my case, I have workloads like this where it’s possible to verify the correctness of the result after inference, so any success rate is better than 0 as it’s possible to identify the “good ones”.
Re:Gemini 2.5 Flash by nonethewiser on Thursday April 17, 2025 @08:10PM (#43723612)
Aren’t you basically just saying you are able to measure the error rate? I mean that’s good, but already a given in this scenario where hes reporting the 7% error rate.
Re:Gemini 2.5 Flash by jsnell on Thursday April 17, 2025 @09:02PM (#43723902)
No. If you're able to verify correctness of individual items of work, you can accept the 93% of verified items as-is and send the remaining 7% to some more expensive slow path.

That's very different from just knowing the aggregate error rate.

Re:Gemini 2.5 Flash by yjftsjthsd-h on Friday April 18, 2025 @04:31PM (#43731617)
No, it's anything that's harder to write than verify. A simple example is a logic puzzle; it's hard to come up with a solution, but once you have a possible answer it's really easy to check it. In fact, it can be easier to vet multiple answers and tell the machine to try again than solve it once manually.