Bringing K/V context quantisation to Ollama

Re:Bringing K/V context quantisation to Ollama by throwaway314155 on Thursday December 05, 2024 @02:44AM (#42325923)

> Should work well if you have 64G vRAM to run SOTA models locally.

Does anyone have this?

edit: Ah, it's a Mac app.

[ Parent | on HN ]

Re:Bringing K/V context quantisation to Ollama by paradite on Thursday December 05, 2024 @04:21AM (#42326441)

Yeah Mac eats Windows on running LLMs.

My app does support Windows though, you can connect to OpenAI, Claude, OpenRouter, Azure and other 3rd party providers. Just running SOTA LLMs locally can be challenging.

[ Parent | on HN ]

Re:Bringing K/V context quantisation to Ollama by throwaway314155 on Thursday December 05, 2024 @11:04PM (#42336225)

I'm pretty satisfied with my linux nvidia gpu setup. I may not have as much memory on my card, but the speed is almost certainly competitive if not outright faster. Further there are lots of techniques that mitigate this issue like offloading/streaming layers in as needed, quantizing, etc.

It also handles actual training/finetuning better.

[ Parent | on HN ]

Re:Bringing K/V context quantisation to Ollama by gzer0 on Thursday December 05, 2024 @06:21AM (#42327075)

M4 Max with 128 GB RAM here. ;) Love it. A very expensive early Christmas present.

[ Parent | on HN ]