I thought this too, but it sounds like the performance of the GPU ultimately holds it back. Maybe it is the software, but I have yet to see a test on Mac silicon that performed well.
It is not too bad, 7B is doing 432 tokens per second on a Macbook[1].
[1] Video: https://huggingface.co/datasets/huggingface/documentation-im...
Apparently "Hugging Face" have some internal swift code that works (but it has not been released). I'm keen to see how it performs on a maxed out Mac Studio (with all that unified memory available).