News for hackers. Stuff that matters.

A comment on Ask HN: Which LLMs can run locally on most consumer computers.

Re:Ask HN: Which LLMs can run locally on most consumer compu... by bryanlarsen on Wednesday May 22, 2024 @01:30PM (#40443647)
Related question: what's the minimum GPU that's roughly equivalent to Microsoft's Copilot+ spec NPU?

I imagine that Copilot+ will become the target minimum spec for many local LLM products and that most local LLM vendors will use GPU instead of NPU if a good GPU is available.

[ Parent | on HN ]
Re:Ask HN: Which LLMs can run locally on most consumer compu... by kevinkeller on Wednesday May 22, 2024 @01:40PM (#40443763)
The NPU in the Snapdragon SoC used by the Windows Surface laptops was quoted to be ~ 40 trillion ops/s (TOPS).

Nvidia 4070 Ti has roughly the same performance: https://www.techpowerup.com/gpu-specs/geforce-rtx-4070-ti.c3...

Of course, I'm massively oversimplifying, but it should be in the ballpark.

[ Parent | on HN ]
Re:Ask HN: Which LLMs can run locally on most consumer compu... by artemisart on Wednesday May 22, 2024 @05:37PM (#40446933)
No, the Nvidia 4070 Ti has much higher performance, TOPS is for integer operations, the 4070 Ti has ~40 float32 TFLOPS and 641 TOPS https://www.nvidia.com/fr-fr/geforce/graphics-cards/40-serie... (which I would say would be peak TOPS for int4 operations, comparing it to the 4080 datasheet, and a bit more than half that for int8 operations) https://images.nvidia.com/aem-dam/Solutions/geforce/ada/nvid... page 34. I did not find the datasheet for 4070 Ti.
[ Parent | on HN ]
Re:Ask HN: Which LLMs can run locally on most consumer compu... by tda on Wednesday May 22, 2024 @01:40PM (#40443767)
I was looking out for a new laptop but was wondering the same. This NPU thing might be one of Microsoft's bets that pays off, and makes all pre-NPU hardware obsolete quickly. Though of course they have doubled down on various failed projexts before (arm Windows, windows phones, etc)
[ Parent | on HN ]
Re:Ask HN: Which LLMs can run locally on most consumer compu... by imtringued on Thursday May 23, 2024 @02:40AM (#40451244)
Basically any GPU with at least 32GB RAM and 12 TFLOPs.
[ Parent | on HN ]