News for hackers. Stuff that matters.
Re:Show HN: Pi Co-pilot – Evaluation of AI apps made easy by diggan on Thursday May 22, 2025 @03:40PM (#44065966)
> [1] https://withpi.ai/project/Xxyhrg2UR8kZHeNmbdV3

Taking a quick look at this, and particularly the "Bad example" of a Show HN post:

> Title: Show HN: Something Cool - Hey everyone, I made something cool. It's a tool that does stuff. I think it could be useful for some people. Let me know what you think.

It gets a 4.0 (out of 5.0) for "Relevance To Community", which seems strange given there is no info given about the project, so it shouldn't be able to score the relevance at all (I think?). It doesn't give a score at all to "Personal Project Verification" but the author does state "I made ...". It also gives "Formatting Quality" a 3.6 which I think would indicate some formatting issues, but seems properly formatted given the context.

Overall, I'm not sure it's either the model, prompt or particular scorer/evaluator, but something seems off, and maybe a bit more spot checking has to be done to ensure it's accurate.

There are no comments on this comment yet.