But I just don't see the need to mislead and suggest that these models are "on par" with ChatGPT or something like that. They just aren't.
Imagine what a math community could train, they just need access to the model and soem GUI software that can help them train.
So llama based Chat stuff is not yet comparable with ChatGPT but there are already lot of progress made. At this moment coding and math is bad in llama based but other stuff is great, like story creation, also I only could test 3-b 4bit and it is good enough to for example provide me a complex response in valid JSON format.
Then hardware will be a separate business. I think Apple might be caught off-guard by Nvidia on hardware. The latest NVlink and 400Gbps interconnects when combined with H100 next iterations and also rumors of advanced PCIe motherboards with high lane Nvidia CPUs and it looks to me that next year they can be selling $100-300K physical systems optimized for LLM inference that physically remind me of mainframes.
It is kind of idiotic that some scientist can spend years and a lot of public money to create some technology and then bilionairs are miliking all the profits.
I’m really hoping there are viable distributed and somewhat decentralized eventually consistent training algorithms we could all run in a P2P system. That would be super cool.
However I can easily see that now the framework has been established if a company builds a proprietary curated dataset for specific skills and then pays to spend resources on specialized reinforcement training.
Then they can commercialize that I would think. As people would pay for an LLM that does XYZ the best. Kinda like your Disney example but I was thinking engineering tasks in my head.
Llama has the potential to reach ChatGPT it needs tunning to get better at responding to questions, llama if I am not worng is mostly attempting to predict what is next.
I can see it similar like Midjourney and Stable diffusion, midjorny can make any stupid prompt look like a digiatal art in the style of Midjorney but look how many stable Diffusion innovation happens, a competent person that is on top with all the new stuff can produce absolute anything in any style they want.