You need ~24-26GB for basic setup (17-19GB model + 128K 8 bit context), so you can, but not much memory would be left for doing anything else on that machine. And even then it would run at like 5-10 t/s (due to relatively low memory bandwidth of M3 Pro) and slow prompt processing (couple hundreds of t/s?)

If they end up releasing updated 35B-A3B variant, then it would be much more interesting in generation speed (~50 t/s)

For inference engine/server you have two (main) choices: llama.cpp for platform-agnostic, MLX for Apple-only. They will spin up OpenAI-compatible local server, and you point your agent harness to it.

For llama.cpp this should be reasonable (maybe shrink context to 128K) starting point: https://x.com/ggerganov/status/2088312671196082312

So you can use llama.cpp and point a harness to it? What harness do you use?

i have this exact same machine and 3.8-27b runs well, albeit slow at about 6 t/s.

i have it complete tasks in the background as i work on other things.

for comparison, my 3090 runs it at about 60 t/s.

What are you using to run the model and as coding harness?

llama.cpp + pi.dev