News for hackers. Stuff that matters.
Re:IBM CEO says open weight models are now irresistibly good... by Gareth321 on Monday October 26, 2026 @07:07AM (#50241882)
Thanks for the tip! I've already got a home server set up. Maybe this is my next AI purchase. Still, US$2,600 here in Europe is not cheap :-/
Re:IBM CEO says open weight models are now irresistibly good... by jacquesm on Monday October 26, 2026 @10:28AM (#50244346)
That's true. Now let's reverse things a bit, we go back five years in time and I give you 'magic in a box' for $2600. Would you buy it?
Re:IBM CEO says open weight models are now irresistibly good... by ThunderSizzle on Tuesday October 27, 2026 @06:11AM (#50256850)
I was hesitant, so I went with a hassle-free return route when I bought mine.

My stack uses a local llamacpp ROCM build, modified by https://github.com/stew675/llama-cpp-rdna-boosts

The build uses my machine's ROCM and I just link it instead of embedded all that in an image, like the published ones do.

Then deploying it via docker compose to my local machine. The models I want to run are stored in a docker volume, so it's easy to switch the model while keeping the same build.

The biggest negative for me is that I don't think it's enough to do parallelization through llamacpp (I'd prefer to work on one thing faster than two things slowly). I haven't gotten vllm working yet. Also, once you have one, you constantly start thinking about a second card...