Ask HN: Some tips for evaluating intelligent agents?

Posted by bruturis on Monday March 25, 2024 @08:05AM
from the ask-hn dept.

A friend of mine is a solo developer, he is creating a big intelligent actors platform using LLMs. I think his platform is overly abstract and use a lot of calls to LLMs. How can one measure the increase in intelligent behavior of this platform versus vanilla GPT4?, I am thinking in same use case that would allow him to show the strength of his idea without having a huge cost.

( 5 comments | on HN )

Re:Ask HN: Some tips for evaluating intelligent agents? by aristofun on Monday March 25, 2024 @10:15AM (#39816616)

How do you expect to measure something abstract that is not yet even defined

[ Parent | on HN ]

Re:Ask HN: Some tips for evaluating intelligent agents? by bruturis on Monday March 25, 2024 @10:58AM (#39817140)

By solving a hard problem or providing a new way of attacking some problem an agent can show behavior that mimic intelligence. What kind of problem should this system attack?

[ Parent | on HN ]

Re:Ask HN: Some tips for evaluating intelligent agents? by aristofun on Monday March 25, 2024 @01:13PM (#39818832)

That is my point - you start from the wrong end. You first choose the specific problem and then measure against that problem.

[ Parent | on HN ]

Re:Ask HN: Some tips for evaluating intelligent agents? by willd13 on Monday March 25, 2024 @01:22PM (#39818953)

What do you mean by an "intelligent actors platform"?

[ Parent | on HN ]

Re:Ask HN: Some tips for evaluating intelligent agents? by bruturis on Monday March 25, 2024 @03:14PM (#39820214)

The platform allows the agents to create new tools and organize knowledge. Is about 60000 lines of code, about six months 10 hours/day seven days a week. I hope my friend get the appropriate funding to continue with his project, as karpathy recommend in one of his talk, first get the system to obtain top capacity, then reduce costs.

[ Parent | on HN ]