I collected notes on the lowest cost hosted LLMs from the major vendors when I wrote up Amazon Nova last week: https://simonwillison.net/2024/Dec/4/amazon-nova/
Nova Micro is $0.035/$0.14 and Google's Gemini 1.5 Flash 8B is $0.0375/$0.15 - just beating those OpenRouter prices, but it may well be that the Qwen models provide better results.
Also worth shouting out you can get Meta's latest llama-3.3:70b (comparable to llama3.1:405b but must faster and cheaper) within GroqCloud's free quotas running at an impressive 276 tok/s.
Right now, with the availability of open weights for cutting-edge models, it feels like this wave of technological advance is pleasantly decentralised however. I can download and run a model and tinker with things which at least feel like the seeds of such a future, where I _might_ be able to build things with my own interests at heart.
But what happens if these models stop being shared, and how likely is that? Reading about the vast quantities of compute deployed to train them, replicating the successes of the main players with a community of volunteers just seems an order of magnitude less achievable than traditional OSS efforts like Linux. This wave feels so tied to massive scale for its success, what do we do if big-tech stop handing out models?
But I don't see why Meta and Alibaba would stop releasing their best models as OSS, since they benefit from the tooling, optimizations and software ecosystems being developed around their OSS models and don't benefit from a future where the best AI models are centralized behind the big tech Corps. As long as their core business remain profitable I don't expect them to stop improving and sharing their OSS models.
Having a personal robot would be great, but they have to invent a fully offline real positronic brain before I will consider allowing one in my house.
Fully open source might be too much to hope for, but that would obviously be the ideal. If it is closed source it definitely should be offline. I can have another, carefully sandboxed, AI in my computer that can help out with tasks that require online access. No need for the two types to be built into the same device.
Prediction 2: The ecosystem around open source models will grow to be much larger, richer, and deeper than closed source models.
If these are true, then OpenAI and Anthropic are in a precarious place. They basically burned a lot of capital to show the open source second movers what to build.
It's also great to see qwq's open chain of thought baked in a OSS LLM so you can see it reason with itself in real-time, it's the kind of secret sauce that proprietary LLMs like o1 would prefer to keep hidden to try build a moat around.
We've got a lot to thank Meta and Qwen for in continually releasing improving high quality OSS models which also encourages others to follow. High quality OSS models are the best thing keeping the cost of LLMs down, you can get unbelievable value on OpenRouter with qwen2.5-coder:32b at $0.08/$0.18 M/tok qwq:32 available at $0.15/$0.60 M/tok which is more than 18x cheaper than Anthropic's latest budget Haiku 3.5 model at $0.80/$4 M/tok (4x price hike over Haiku 3.0).