News for hackers. Stuff that matters.

A comment on Sam Altman Seeks Trillions of Dollars to Reshape Business of Chips and AI. Parent comment

Re:Sam Altman Seeks Trillions of Dollars to Reshape Business... by yazzku on Thursday February 08, 2024 @11:00PM (#39311234)
Can you link to an actual performance analysis on this?
[ Parent | on HN ]
Re:Sam Altman Seeks Trillions of Dollars to Reshape Business... by simne on Friday February 09, 2024 @11:07PM (#39323491)
Easy. I made tests on desktop core i7-7700 with 64G DDR4-2400. And I've tested 13B..30B..70B models on it, and you may imagine, how easy to manage how many CPU cores used.

Answer is - it is really works, but slow (about 0.5..1 tokens per second, with near 100% CPU usage).

i7-7700 is good weighted machine, but before I few times achieved memory speed bounds with highly optimized software. And it looks very different. When use all cores, I got somewhere about 50% of CPU usage.

BTW Llama.CPU is very good software.

[ Parent | on HN ]