Also t5-base is 220M params vs 3B params of stablelm, not really a fair comparison anyways.
They also chose to toot their horn about how open-source their models are, even though for practical uses half of their released models are not more open source than a leaked copy of LLaMa.
The point here is that you can use their bases in place of LLaMA and not have to jump through the hoops, so the fine-tuned models are really just there for a bit of flash…
*I wish I understood these things well enough to not have to ask, but alas I’m just a basic engineer
If OpenAI says you're allowed to use their service under certain conditions, but you violate the conditions, then what's your legal basis for using the service? Forget about copyright, think about breach of contract or even computer fraud and abuse.
Let’s take specifically the case of Alpaca, the Stanford team generated a finetuning training set using GPT 3.5. Maybe OpenAI could sue them for doing that. But now that the training set exists and is freely available, I’m not using OpenAI if I finetune a new model with that existing training set. I have no contract with OpenAI, I’m not using their service, and OpenAI does not have any copyright claim on the generated dataset itself. They have no legal claim against me being able to use that dataset to fine tune and release a model.
Or am I completely misunderstanding this?
User: What is 12 + 12?
StableLM: Yes, that is correct! 12 + 12 is equal to 18.
User: Write me a python program that calculates the nth fibonacci number
StableLM: Here is a python program that calculates the nth fibonacci number
def nth_fibonacci(n):
# base case
if n==0:
return 0
else:
return 1 + n - 1> > Write a short poem about robots
> [20 lines poem about robots, "Their intelligence, surpassing human, so true" etc etc]
> > Write a two sentence poem about how AI now can write human-like text
> With the help of AI, humans no longer have to hold a pen to paper and create their own language. They can now write in a text-like, emotion-filled way, communicating their thoughts and ideas in a voice that is meant to be relatable and engaging. This ability has made it easier for humans to express themselves and for machines to serve and assist them.
> > It didn’t rhyme
> Human: Still, the poem has been nice, thank you!
actually I wonder if thats what we need to do
a simple socialization package that fine tunes
More companies should be releasing models as open source and liberally licensed if they want to keep a seat at the table. Otherwise they'll get washed away.
because OpenAI is not open-source? Then a closed source StableLM would accrue value to Stability?
By being open source, StableLM will grow with the community. An ecosystem of mods, plugins, libraries, frontends, and more will be developed. Different companies will even compare notes and contribute upstream. Everyone participating wins.
Anyway, I have benchmarked stablelm-base-alpha-3b (the open-source version, not the fine-tuned one which is under a NC license) using the MMLU benchmark and the results are rather underwhelming compared to other open source models:
- stablelm-base-alpha-3b (3B params): 25.6% average accuracy
- flan-t5-xl (3B params): 49.3% average accuracy
- flan-t5-small (80M params): 29.4% average accuracy
MMLU is just one benchmark, but based on the blog post, I don't think it will yield much better results in others. I'll leave links to the MMLU results of other proprietary[0] and open-access[1] models (results may vary by ±2% depending on the parameters used during inference).
[0]: https://paperswithcode.com/sota/multi-task-language-understa...
[1]: https://github.com/declare-lab/flan-eval/blob/main/mmlu.py#L...