Re:Serving AI from the Basement – 192GB of VRAM Setup
by mysteria on Sunday September 08, 2024 @03:09PM
(#41482392)
The 3090s also give you access to all those CUDA inference and training frameworks used in industry. With Apple you're pretty much stuck with either MLX or llama.cpp.
Honestly though I'd be curious to see a cost analysis of Apple vs. Nvidia for commercial batched inference. The Nvidia system can obviously spit out more tokens/s but for the same price you could have multiple Mac Studios running the same model (and users would be dispatched to one of them).
Honestly though I'd be curious to see a cost analysis of Apple vs. Nvidia for commercial batched inference. The Nvidia system can obviously spit out more tokens/s but for the same price you could have multiple Mac Studios running the same model (and users would be dispatched to one of them).