News for hackers. Stuff that matters.

A comment on Building a personal, private AI computer on a budget. Parent comment

Re:Building a personal, private AI computer on a budget by CamperBob2 on Tuesday February 11, 2025 @10:16AM (#43013759)
What I'd like to know is how well those dual-Epyc machines run the 1.58 bit dynamic quant model. It really does seem to be almost as good as the full Q8.
[ Parent | on HN ]
Re:Building a personal, private AI computer on a budget by DrNosferatu on Wednesday February 12, 2025 @05:04PM (#43030238)
I tried that that: ~1.5 to 3 tokens/sec.
[ Parent | on HN ]
Re:Building a personal, private AI computer on a budget by CamperBob2 on Thursday February 13, 2025 @08:13PM (#43043531)
Ouch, thanks. About what I get now on a single-CPU box with 128 GB+a 4090. Was hoping for a major speedup.
[ Parent | on HN ]
Re:Building a personal, private AI computer on a budget by DrNosferatu on Friday February 14, 2025 @05:58PM (#43053990)
Peak performance is achieved at ~21 cores. Bottleneck - without any special configs - is RAM to CPU bandwidth.

Let me know if you find some config that really leverages more cores!

[ Parent | on HN ]