I've had some rough time getting LM-Studio properly configured for multi-card. It exists, but I feel like it is kind of buggy. I will disable a card and it will still load the model into it. Sometimes it will split the model even though there is loads of room available.

I might need to finally make the switch away from it, but it is so convenient, especially as a chat interface for system prompt experimentation.

vLLM is probably the key difference there… its scheduler is built around batching/concurrency, while this setup is heavily optimized llama.cpp for single-stream latency