News for hackers. Stuff that matters.
Re:Show HN: Phind Model beats GPT-4 at coding, with GPT-3.5 ... by pleonasticity on Tuesday October 31, 2023 @02:11PM (#38088995)
This is great work, but HumanEval is an extremely limited benchmark and I don’t think you can seriously claim to beat GPT-4 at coding based only on that metric.
Re:Show HN: Phind Model beats GPT-4 at coding, with GPT-3.5 ... by nomel on Tuesday October 31, 2023 @02:23PM (#38089181)
Fifth sentence:

> However, we’ve found that HumanEval is a poor indicator of real-world helpfulness.

Re:Show HN: Phind Model beats GPT-4 at coding, with GPT-3.5 ... by rushingcreek on Tuesday October 31, 2023 @02:12PM (#38089016)
Thank you. You're right -- which is why we rely on feedback we've received from our own users for that claim. Many of our users who have the choice to use either GPT-4 or the Phind Model on Phind choose the Phind Model.
Re:Show HN: Phind Model beats GPT-4 at coding, with GPT-3.5 ... by Kranar on Tuesday October 31, 2023 @02:15PM (#38089076)
You likely know this, but keep in mind the kind of selection bias in taking feedback mostly from your own users. The number of times I've heard product designers claim that their users prefer some aspect of how their application already works, ignoring the fact that the users who didn't prefer it have left and hence are likely not available to survey.
Re:Show HN: Phind Model beats GPT-4 at coding, with GPT-3.5 ... by rushingcreek on Tuesday October 31, 2023 @02:18PM (#38089107)
Of course. We do our best to talk to churned users as well, but we're doing this Show HN to get even more diverse feedback.
Re:Show HN: Phind Model beats GPT-4 at coding, with GPT-3.5 ... by pleonasticity on Tuesday October 31, 2023 @02:14PM (#38089054)
I understand, but big claims require big evidence and so it’s still IMHO not rhetorically a strong position. I’m glad people find it more useful!