It, of course, runs on linux. I went with stock ubuntu so I could attach a screen and/or remote in to a full GUI. Having a full GUI also makes the rig more relatable to people who's eye glaze over when they see a terminal interface. "It's just a funny-looking desktop computer."
At 48GB of VRAM it can run basically any LLM I would want to download. Speeds are fine for a single user (Tok/s numbers change based on model/mode). But if I were using it for something like ComyfUI I would ditch the multiple 5060s and go with a 5070ti or 5080 as ComfyUI does better with a single powerful GPU.
I'm sure cloud models will always have a slight edge but most tasks, most of the time, won't need that few extra percentage points of performance.
Llama 3.1 8B baked into an ASIC. Expect 1/10th the power requirements, and other savings.
This is one of the reasons I don't believe the "it'll never be profitable" narrative. Just stop training when you think your model's "good enough" and move to custom silicon.
Yes and no. I bought most of the hardware before the rampocalypse. It is just an AM4 chip on a nothing motherboard. Even today, 16GB 5060s are maybe 500 each. 64GB of DDR4 is another 500$. Add a power supply and pci risers and the basic parts could be purchased today for around 3000$... call it 4000$ all-in including my labor to get everything connected and talking to each other.
4000$ is money but is also almost nothing in the world of commercial IT purchases.
(People actually came by my desk to see "the AI", the physical thing. It blows people's minds to realize AI doesn't need a data center, that it is just some computer parts sitting on a desk.)
The big hitter AI companies see the writing on the wall: open source models and local AI rigs will decimate the market for big AI in the next few years.