Re:Mistral 7B,The complete Guide of the Best 7B model
by brucethemoose2 on Monday October 30, 2023 @11:27PM
(#38079917)
No offense, but at first glance I thought this was a GPT generated SEO ad farm... with titles like "Best AI chat bot Zephyr 7B inside finetuning Mistral 7b," "How to Finetune mistral 7B for free and easily with AutoTrain" and the general format.
But its ad free, with no promotional links, and the information is good. For instance, I like the VRAM calculator, and the list of the top finetunes is excellent (like Zephyr Beta and SciPhi, both of which are very new and quite impressive).
Mistral is indeed amazing. I hope they keep updating it, and maybe even publish a 13B version.
Re:Mistral 7B,The complete Guide of the Best 7B model
by brucethemoose2 on Tuesday October 31, 2023 @02:07AM
(#38080881)
Its weird... it has AI generated images for the finetunes listings as well, complete with some text, with descriptions taken from HF of course. But some of it is... off. Seemingly not human curated.
And of course the Mistral company page does not list any finetunes.
Is this a click farm? Bait and switch for future ads? What the heck is this?
Re:Mistral 7B,The complete Guide of the Best 7B model
by behnamoh on Tuesday October 31, 2023 @01:13AM
(#38080602)
IIRC there was a recent paper that showed Mistral may have data contamination, leading to high scores on the benchmarks. In my experiences with Zephyr, OpenOrca, and Dolphin finetunes of Mistral, the model did demonstrate ChatGPT-esque format in responses, but it didn't demonstrate any more intelligence than vanilla Llama 2 7B.
Another thing to worry about is that Mistral (the company) has publicly said that future larger models will not necessarily be open like the 7b model. After a certain model lock-in has taken place, the company will guard the model behind API (as they mentioned) and it won't be open anymore.
Re:Mistral 7B,The complete Guide of the Best 7B model
by kcorbitt on Tuesday October 31, 2023 @01:22AM
(#38080651)
I don't know whether Mistral had benchmark data contamination or not, but it's definitely a stronger model than Llama 2 7b or 13b even on non-benchmark tasks.
We proactively tested Mistral across all of our customer-deployed models on our fine-tuning platform (openpipe.ai) and found that with the exact same training data it outperformed both of those models consistently. At this point we're only keeping Llama 2 7b/13b around for our legacy customers and recommend that everyone training new models use Mistral because it's just so much better.
Re:Mistral 7B,The complete Guide of the Best 7B model
by avereveard on Tuesday October 31, 2023 @02:00AM
(#38080841)
I've tested it and many fine tunes and while it seems to work ok for short chat it's really inattentive in summarisation and information extraction tasks. Continuation of long context stories/chat is also severely lacking, making up lot of contradictory details as it goes on.
It might or might not be better in benchmarks, but I just don't see it in practice for many tasks where you'd pick a LLM and go
There is one thing tho where I've seen it perform really well, which is single task fine tuning. It's small enough you can do full tuning and it's steerable enough you don't need more than few thousand samples.
Re:Mistral 7B,The complete Guide of the Best 7B model
by brucethemoose2 on Tuesday October 31, 2023 @02:15AM
(#38080936)
Eh, I've found llama 7b finetunes to be underwhelming too. They both seem to fall apart at detail retrieval and such, even with Mistral's longer base context.
For what its worth, Mistral finetunes excel in this ranking, and Mistral AI almost certainly did not contaminate it with that test: https://rentry.co/ayumi_erp_rating
Re:Mistral 7B,The complete Guide of the Best 7B model
by BrunoJo on Tuesday October 31, 2023 @01:25AM
(#38080665)
We are using Mistral at https://lemonfox.ai/. With Lemonfox you can easily switch from the OpenAI API to open source AI models.
Mistral has been working well as a gpt-3.5 replacement but we are currently planning to switch to Zephyr - a fine-tuned Mistral model. We've seen some nice improvements with Zephyr especially with non-English prompts/responses.
But its ad free, with no promotional links, and the information is good. For instance, I like the VRAM calculator, and the list of the top finetunes is excellent (like Zephyr Beta and SciPhi, both of which are very new and quite impressive).
Mistral is indeed amazing. I hope they keep updating it, and maybe even publish a 13B version.
You're not far off. The majority of the content is ripped from the real site: https://mistral.ai/news/announcing-mistral-7b/ and https://docs.mistral.ai/usage/how-to-use/
Unfortunately the real site doesn't have gems like "Top rated Vedio about Mistral 7B " in 30pt italics.
And of course the Mistral company page does not list any finetunes.
Is this a click farm? Bait and switch for future ads? What the heck is this?
my site include: it contains:
- What is Mistral 7B? - Can I run it with my GPU? - Top Mistral 7B Fine-tuned Models - Top rated Vedio about Mistral 7B - how to finetune mistral 7b
mistral-7b.com
Another thing to worry about is that Mistral (the company) has publicly said that future larger models will not necessarily be open like the 7b model. After a certain model lock-in has taken place, the company will guard the model behind API (as they mentioned) and it won't be open anymore.
We proactively tested Mistral across all of our customer-deployed models on our fine-tuning platform (openpipe.ai) and found that with the exact same training data it outperformed both of those models consistently. At this point we're only keeping Llama 2 7b/13b around for our legacy customers and recommend that everyone training new models use Mistral because it's just so much better.
It might or might not be better in benchmarks, but I just don't see it in practice for many tasks where you'd pick a LLM and go
There is one thing tho where I've seen it perform really well, which is single task fine tuning. It's small enough you can do full tuning and it's steerable enough you don't need more than few thousand samples.
For what its worth, Mistral finetunes excel in this ranking, and Mistral AI almost certainly did not contaminate it with that test: https://rentry.co/ayumi_erp_rating
Inference only: https://lite.koboldai.net/
https://github.com/PygmalionAI/aphrodite-engine https://github.com/LostRuins/koboldcpp
Picking a model to host is quite a rabbit hole, but I am very interested in SciPhi and Zephyr at the moment.
I havent messed with Petals yet either, but swarm finetuning is very appealing... If thats even implemented atm.
While it seems to be a base model, I'm pretty excited about coming finetunes on this one: https://huggingface.co/SciPhi/SciPhi-Mistral-7B-32k