Anthropic's Haiku Beats GPT-4 Turbo in Tool Use

Re:Anthropic's Haiku Beats GPT-4 Turbo in Tool Use by mritchie712 on Monday April 08, 2024 @02:07PM (#39972249)

Not sure when they implemented this, but ollama now has a JSON mode [0]. Not function calling, but one of the simpler ways to get JSON in a local LLM. I'm using it with `knoopx/hermes-2-pro-mistral:7b-q8_0` and it's worked well for me so far.

    response = ollama.chat(model=OLLAMA_MODEL, 
        messages=[
            {
                'role': 'system',
                'content': system_message,
            },
            {
                'role': 'user',
                'content': user_prompt,
            },
        ],
        format='json',
        options = {
            #'temperature': 1.5, # very creative
            'temperature': 0.0
        }
        )

0 - https://github.com/ollama/ollama/blob/main/docs/api.md#json-...

[ Parent | on HN ]

Re:Anthropic's Haiku Beats GPT-4 Turbo in Tool Use by Joschkabraun on Monday April 08, 2024 @02:08PM (#39972258)

Interesting. Do you have any benchmarks?

[ Parent | on HN ]

Re:Anthropic's Haiku Beats GPT-4 Turbo in Tool Use by mritchie712 on Monday April 08, 2024 @02:11PM (#39972295)

No benchmarks, just my anecdotal experience trying to get local LLM's to respond with JSON. The method above works for my use case nearly 100% of the time. Other things I've tried (e.g. `outlines`[0]) are really slow or don't work at all. Would love to hear what others have tried!

0 - https://github.com/outlines-dev/outlines

[ Parent | on HN ]

Re:Anthropic's Haiku Beats GPT-4 Turbo in Tool Use by Joschkabraun on Monday April 08, 2024 @02:14PM (#39972325)

Ah yes. Have you tried out instructor [0] or Guidance [1]?

[0]: https://github.com/jxnl/instructor/

[1]: https://github.com/guidance-ai/guidance/tree/main

[ Parent | on HN ]