I was just trying to make sure is maximally performant, and did it with MLX because i am running on mac hardware and wanted to be able to run 30b in reasonable time so it can actually autonomously code something. Otherwise there are many ways to do it!

It would be nice if you mentioned it's about apple silicon, and not apple intel computers. They're still ubiquitous nowadays

we're on the 4th generation of silicon now

mlx is an alternative model format to GGUF. It executes natively on apple silicon using Apple's AI accelerator, rather than through GGUF as a compute shader(!). It's faster and uses fewer resources on Apple devices.

I believe mlx will allow you to run the models marginally faster (per a recent blog post by @simonw)

Yeah, you don't necessarily need it but it's optimized for Apple Silicon and in my experience feels like it gives slightly better performance than GGUFs. I really need to formally measure that so I'm not just running on vibes!

I for one, am willing to just trust you bro ;)

Yeah I’ll go with Simon’s vibes over most people’s measurements!