Snowflake and the like are training and releasing new models because they intend to integrate the AI into their existing product down the line. Why not use and fine-tune an existing model? Their in-grown model maybe better suited for their product. This can also fail like Bloomberg's financial model being inferior to GPT-4, but these companies have to try.
Not all of them have permissive licenses for whatever the companies may want (or their clients want). Kind of a funny situation where everyone would benefit, but no one wants to burn their money for the greater good.
Interesting you'd say that in a discussion on Snowflake's LLM, no less. As someone who has a good opinion of Databricks, genuinely curious what made you arrive at such a damning conclusion.
Yes. Great choice of words. A lot of non-frontier models look like "an exercise of pure marketing" to me.
Still, I fail to see the rationale for telling the world, "Look at us! We can do it too!"
There are DOZENS of orgs releasing foundational models, not "a handful."
Salesforce, EleuthierAI, NVIDIA, Amazon, Stanford, RedPajama, Cohere, Mistral, MosaicML, Yandex, Huawei StabilityLM, ...
https://docs.google.com/spreadsheets/d/1kT4or6b0Fedd-W_jMwYp...
It's completely bonkers and a huge waste of resources. Most of them will see barely any use at all.
Furthermore, I'm happy to be in a golden age with lots of orgs trying things and many options. It's going to suck once the market eventually consolidates us and we have to take whatever enshittified thing the ologopolists feed us.
I want someone to spend a million on a Chess LLM so we can get a sense of how sophisticated they can get at non-linguistic pattern matching.
I want someone to spend a million on an LLM trained on Python program traces so we can try to teach it cause and effect and "debugging". Maybe it will emulate a Python interpreter and get highly reliable at predicting the outcome of Python code.
etc.
In the grand scheme of things, though, most of these are quite small -- 7b range. A 7b model is nothing to sneeze at but it's not megacorp resources either. It's in the range of "VC check" size.
The "big boys" who are training 70b plus are FAANG or government-scale entities. Microsoft, Google, and Meta have multiple entries on that "big" LLM foundation list -- it's because the GPUs are already bought, have to train something to keep utilization up. Also bear in mind that training of these things is still something closer to an art than a science; you put terabytes of data into the cauldron, let it brew, and only after it's done can you taste what you've made. Makes sense that some of these models will be junk.
Did you have a script to generate it or was it manually done?
A group at the CRFM maintains a bigger list of models (their goal is stated for cataloguing foundation models, but it looks like they have some tunes mixed in these days): https://crfm.stanford.edu/ecosystem-graphs/
This site also seems to keep track of models, with more closed/announced models that I don't bother to track: https://lifearchitect.ai/models-table/
There is basically the big guys that everyone's heard of (google, meta, microsoft/openAI, and anthropic) and then a handful of smaller players who are training foundation models mostly so that they can prove to VCs that they are capable of doing so -- to acquire more funding/access to compute so that they may eventually dethrone openAI and take a piece of the multi-billion dollar "enterprise AI" market for themselves.
Below that, there is a frothing ocean of mostly 7B finetunes created mostly by individuals who want to jailbreak base models for... reasons, plus the occasional research group.
The most oddball one I have seen is the databricks LLM which seems to have been an exercise of pure marketing. Those I suspect will disappear when the bubble deflates a bit.