News for hackers. Stuff that matters.
Re:Web LLM – WebGPU Powered Inference of Large Language Models by Tepix on Sunday April 16, 2023 @02:56PM (#35593167)
GTPQ has been the missing piece, it allows quantizing the model weights from 16 to 4 bits with only a small loss in quality. That it turn allows running even the large 65 billion parameter version of the LLaMA model in ~33GB of RAM or VRAM.

With VRAM that requires two 24GB GPUs which is no longer completely out of reach.

The model running in the browser is a smaller version with 7 billion parameters, which is good enough for some things.

There are no comments on this comment yet.