Mistral 7B, the most powerful language model for its size to date, Apache 2.0

Re:Mistral 7B, the most powerful language model for its size... by Karrot_Kream on Wednesday September 27, 2023 @02:50PM (#37679411)

How many tokens / sec are you getting on an M1 Air? Curious since I'm at work and can't try this on my Air yet hah.

[ Parent | on HN ]

Re:Mistral 7B, the most powerful language model for its size... by brucethemoose2 on Wednesday September 27, 2023 @03:39PM (#37680123)

Its the same speed as llama 7B, so very quick.

[ Parent | on HN ]

Re:Mistral 7B, the most powerful language model for its size... by Patrick_Devine on Wednesday September 27, 2023 @03:12PM (#37679727)

I'm getting >30 tokens/sec using it with ollama and an M2 Pro. That might be a little slow though because I have a background finetuning job running.

[ Parent | on HN ]

Re:Mistral 7B, the most powerful language model for its size... by minzi on Wednesday September 27, 2023 @06:45PM (#37682463)

Bit of a tangential question here, but any recommendations on how to get started fine tuning this model (or ones like it)? I feel like there are a million different tutorial and ways of doing it when I google.

[ Parent | on HN ]

Re:Mistral 7B, the most powerful language model for its size... by Karrot_Kream on Wednesday September 27, 2023 @08:09PM (#37683397)

Take a look at the QLoRA repo https://github.com/artidoro/qlora/ which has an example finetuning Llama. Made by the authors of the QLoRA paper.

[ Parent | on HN ]

Re:Mistral 7B, the most powerful language model for its size... by brucethemoose2 on Wednesday September 27, 2023 @09:17PM (#37683950)

https://github.com/OpenAccess-AI-Collective/axolotl

This is a wrapper around many training methods, and it has yielded many excellent community finetunes already.

[ Parent | on HN ]

Re:Mistral 7B, the most powerful language model for its size... by anonyfox on Wednesday September 27, 2023 @02:58PM (#37679528)

feels roughly like the same speed as GPT3.5 in the browser UI

[ Parent | on HN ]

Re:Mistral 7B, the most powerful language model for its size... by benkaiser on Wednesday September 27, 2023 @09:22PM (#37683982)

~13 tokens per second. I asked for a story about goldilocks and this was the timings on my M1 air using `ollama run mistral --verbose`:

total duration: 33.964492834s

load duration: 1.471584ms

prompt eval count: 8 token(s)

prompt eval duration: 385.334ms

prompt eval rate: 20.76 tokens/s

eval count: 418 token(s)

eval duration: 31.887033s

eval rate: 13.11 tokens/s

[ Parent | on HN ]