Everything around LLMs is still magical and wishful thinking

Re:Everything around LLMs is still magical and wishful thinking by wubrr on Friday July 04, 2025 @06:10PM (#44468291)

I'm no LLM evangelist, far from it, but I expect models of similar quality to the current bleeding-edge, will be freely runnable on consumer hardware within 3 years. Future bleeding-edge models may well be more expensive than current ones, who knows.

[ Parent | on HN ]

Re:Everything around LLMs is still magical and wishful thinking by ls612 on Saturday July 05, 2025 @12:50AM (#44470149)

How do the best models that can run on say a single 4090 today compare to GPT 3.5?

[ Parent | on HN ]

Re:Everything around LLMs is still magical and wishful thinking by electroglyph on Saturday July 05, 2025 @01:21AM (#44470251)

Qwen 2.5 32B which is an older model at this point clearly outperforms it:

https://llm-stats.com/models/compare/gpt-3.5-turbo-0125-vs-q...

[ Parent | on HN ]

Re:Everything around LLMs is still magical and wishful thinking by ls612 on Saturday July 05, 2025 @01:30AM (#44470287)

Even when quantized down to 4 bits to fit on a 4090?

[ Parent | on HN ]

Re:Everything around LLMs is still magical and wishful thinking by FuckButtons on Saturday July 05, 2025 @06:35PM (#44476135)

Not in my experience, running qwen3:32b is good, but it’s not as coherent or useful as 3.5 at a 4bit quant. But the gap is a lot narrower than llama 70b.

[ Parent | on HN ]

Re:Everything around LLMs is still magical and wishful thinking by TeMPOraL on Saturday July 05, 2025 @03:02AM (#44470662)

For the purpose of keeping the costs of LLM-dependent services down, you don't need to run bleeding-edge models on single consumer GPUs. Even if it takes a hundred GPUs, it still means people can start businesses around hosting those models, and compete with the large vendors.

[ Parent | on HN ]