How to run Qwen 3.5 locally

Re:How to run Qwen 3.5 locally by teaearlgraycold on Sunday March 08, 2026 @03:41AM (#47295404)

Qwen3.5 35B A3B is much much faster and fits if you get a 3 bit version. How fast are you getting 27B to run?

On my M3 Air w/ 24GB of memory 27B is 2 tok/s but 35B A3B is 14-22 tok/s which is actually usable.

[ Parent | on HN ]

Re:How to run Qwen 3.5 locally by ece on Sunday March 08, 2026 @04:03AM (#47295501)

The 27B is rated slightly higher for SWE-bench.

[ Parent | on HN ]

Re:How to run Qwen 3.5 locally by throwdbaaway on Sunday March 08, 2026 @05:01AM (#47295741)

Using ik_llama.cpp to run a 27B 4bpw quant on a RTX 3090, I get 1312 tok/s PP and 40.7 tok/s TG at zero context, dropping to 1009 tok/s PP and 36.2 tok/s TG at 40960 context.

35B A3B is faster but didn't do too well in my limited testing.

[ Parent | on HN ]

Re:How to run Qwen 3.5 locally by ranger_danger on Monday March 09, 2026 @01:03AM (#47305058)

with regular llama.cpp on a 3070ti I get 60tok/s TG with the 9B model, it's quite impressive.

[ Parent | on HN ]

Re:How to run Qwen 3.5 locally by ranger_danger on Monday March 09, 2026 @12:58AM (#47305022)

Don't sleep on the 9B version either, I get much faster speeds and can't tell any difference in quality. On my 3070ti I get ~60tok/s with it, and half that with the 35B-A3B.

[ Parent | on HN ]

Re:How to run Qwen 3.5 locally by andai on Wednesday March 11, 2026 @03:53PM (#47340408)

27B needs less memory and does better on benchmarks, but 35B-A3B seems to run roughly twice as fast.

[ Parent | on HN ]