News for hackers. Stuff that matters.
Re:Ollama is now available on Windows in preview by imtringued on Sunday February 18, 2024 @05:38AM (#39417963)
To be fair a lot of the GPU edge comes from fast memory. A GPU with 20tflops running a 30 billion parameter model has a compute budget of 700flops per parameter. Meanwhile the sheer size of the model prevents you from loading it more than 20 times from memory per second.

There are no comments on this comment yet.