Sort of works for inference, but not for anything else. For now, I bought a 4x4090 workstation… but I am crying how power-inefficient it is.

For my education, would you be able to expand on what it doesn't work for and why? Thank you!

Inference (token generation) is memory-bound, KV cache prefill (prompt processing) is compute-bound. The ARM Macintoshes have lots of memory bandwidth but not a lot of compute power, so they're great for outputting text but terrible for tasks like analyzing documents. I've never done fine-tuning but my understanding is that that is a highly-parallelizable compute hog as well.

You might like this article, which looks at the arithmetic intensity of LLM processing: https://www.baseten.co/blog/llm-transformer-inference-guide/