News for hackers. Stuff that matters.
Re:Llama.cpp Now Supports Qwen2-VL (Vision Language Model) by simonw on Saturday December 14, 2024 @09:48PM (#42421121)
Yeah, Haiku 3.5 doesn't really count as a budget model - Anthropic have kept the original Claude 3 Haiku around for their budget entry.

I collected notes on the lowest cost hosted LLMs from the major vendors when I wrote up Amazon Nova last week: https://simonwillison.net/2024/Dec/4/amazon-nova/

Nova Micro is $0.035/$0.14 and Google's Gemini 1.5 Flash 8B is $0.0375/$0.15 - just beating those OpenRouter prices, but it may well be that the Qwen models provide better results.

Re:Llama.cpp Now Supports Qwen2-VL (Vision Language Model) by mythz on Saturday December 14, 2024 @10:04PM (#42421175)
Yeah I'm currently using Gemini 2 Flash (exp) free quota for a premium hosted model, it's a surprisingly great model, IMO Google has caught up with the leaders with their latest experimental models. I've also tested Nova's models, which are pretty high quality and exception value (lite/micro) for their performance.

Also worth shouting out you can get Meta's latest llama-3.3:70b (comparable to llama3.1:405b but must faster and cheaper) within GroqCloud's free quotas running at an impressive 276 tok/s.

Re:Llama.cpp Now Supports Qwen2-VL (Vision Language Model) by antupis on Sunday December 15, 2024 @05:34AM (#42422724)
Have you tested tool calling capabilities of these cheaper models?
Re:Llama.cpp Now Supports Qwen2-VL (Vision Language Model) by simonw on Sunday December 15, 2024 @07:46AM (#42423145)
Not yet. A lot of them make bold claims and there are benchmarks like https://gorilla.cs.berkeley.edu/leaderboard.html but I don't have my own function calling test harness setup yet.
Re:Llama.cpp Now Supports Qwen2-VL (Vision Language Model) by leobg on Sunday December 15, 2024 @04:35PM (#42426145)
You know that you can make any model call and use tools simply by giving it few shot examples and writing your own parsing logic. I’ve done it many times for clients, both at the prompt and at the fine-tune level.