Anthropic's Haiku Beats GPT-4 Turbo in Tool Use

Re:Anthropic's Haiku Beats GPT-4 Turbo in Tool Use by airocker on Monday April 08, 2024 @02:03PM (#39972210)

Are tools fine tuned in both or just prompt based? For a fair comparison

[ Parent | on HN ]

Re:Anthropic's Haiku Beats GPT-4 Turbo in Tool Use by fudoshin2596 on Monday April 08, 2024 @03:59PM (#39973133)

No fine tuning. Looks like he's do raw model capabilities with simple prompt. repo: https://github.com/parea-ai/tool-use-benchmark

[ Parent | on HN ]

Re:Anthropic's Haiku Beats GPT-4 Turbo in Tool Use by Joschkabraun on Monday April 08, 2024 @02:06PM (#39972238)

All models got the same prompt fed which was essentially "Question: {question}". And then the API's accept the function call definition

[ Parent | on HN ]

Re:Anthropic's Haiku Beats GPT-4 Turbo in Tool Use by airocker on Monday April 08, 2024 @02:12PM (#39972302)

But what if there is a bad model that fine tunes with only tools?

[ Parent | on HN ]

Re:Anthropic's Haiku Beats GPT-4 Turbo in Tool Use by Joschkabraun on Monday April 08, 2024 @02:19PM (#39972379)

What do you mean with being a bad model?

If the model is really good at tool use, then it will broadly useful as it needs capabilities to generate the tool definition. So, there should be some transferability.

[ Parent | on HN ]

Re:Anthropic's Haiku Beats GPT-4 Turbo in Tool Use by ec109685 on Monday April 08, 2024 @10:25PM (#39975761)

If the exact tools test was part of the training of the model, wouldn’t that throw off the results and not generalize?

[ Parent | on HN ]