We proactively tested Mistral across all of our customer-deployed models on our fine-tuning platform (openpipe.ai) and found that with the exact same training data it outperformed both of those models consistently. At this point we're only keeping Llama 2 7b/13b around for our legacy customers and recommend that everyone training new models use Mistral because it's just so much better.
It might or might not be better in benchmarks, but I just don't see it in practice for many tasks where you'd pick a LLM and go
There is one thing tho where I've seen it perform really well, which is single task fine tuning. It's small enough you can do full tuning and it's steerable enough you don't need more than few thousand samples.
For what its worth, Mistral finetunes excel in this ranking, and Mistral AI almost certainly did not contaminate it with that test: https://rentry.co/ayumi_erp_rating
https://github.com/PygmalionAI/aphrodite-engine https://github.com/LostRuins/koboldcpp
Picking a model to host is quite a rabbit hole, but I am very interested in SciPhi and Zephyr at the moment.
I havent messed with Petals yet either, but swarm finetuning is very appealing... If thats even implemented atm.
Another thing to worry about is that Mistral (the company) has publicly said that future larger models will not necessarily be open like the 7b model. After a certain model lock-in has taken place, the company will guard the model behind API (as they mentioned) and it won't be open anymore.