I've been playing with it on the recent iPhone Pro, and it is pretty fast. The mistral 7B is very wordy though and is prone to making up stuff. It's fun to talk to, but it's not very useful currently.

Based on this demonstration of a local LLM, with optimizations, it definitely the way of the future on apple devices at least.