Also do you know what the best local setup is?
edit: found some oudated mesa/radv/shaderc/glslang packages to update, and up to 34 tok/s now. MORE MORE
Looking a bit oof on reddit rn, I'm seeing reports of 10 tok/s via llama.cpp :|
https://www.reddit.com/r/LocalLLaMA/comments/1y1g9cr/inkling...
Edit: Hopeful that this gets the ninfer treatment, if the benchmarks are to be believed it feels like there is probably both amazing speed and real performance to be unlocked by the community here on the x090 cards
OK, but please come back and let us know your results!
Flash Attention (FA) is not supported on the current llama.cpp (at least not for ROCm) for this model so KV cache is forced to f16 for the value (V) and I cannot lower it to q4 go get decent KV cache. But the key (K) could be set to q4, and that got me about 100k of ctx. Current benching shows ~28 tok/s decode and ~481 tok/s prefill.
It's a start, but I expected better speeds even on a non ideal setup. I'll keep experimenting!
I'm seing around 2800t/s on pp at the very beginning of the conversation, and then going to around 800 by the 256k context size...
I've also update the value in the gguf to have instead of context size 256k be triple and have use it all the way to around 600k context size
use the cache eviction branch https://github.com/alainnothere/llama.cpp/tree/disk-cache-ev...
llama-server -m /path_to_model/Qwen3.8-27B-UD-IQ4XS_u.gguf -np 2 --host 0.0.0.0 --port 8089 --kv-unified --temp 0.2 --top-k 20 --min-p 0.0 --cache-disk-path /path_to_store_conversations_to_disk --cache-ram 100 --cpu-range 0-23 --reasoning-budget 4096 --reasoning-budget-message "... I am thinking for too long -- let me gather more info about the task." --no-cache-idle-slots --spec-draft-auto --spec-draft-sampling probabilistic --spec-draft-n-min 1 --spec-draft-p-min 0 --spec-ngram-auto --spec-ngram-mod-size 35544320 --spec-ngram-mod-cache /path_to_storage_of_ngram_mod_this_is_specific_per_model/ngram-mod-qwen3.8-27b.bin --fit on --device Vulkan3,Vulkan2,Vulkan1 --fit-target 1024,2560,2560 --ctx-checkpoints 25 --checkpoint-spill-max 25 --cache-type-k f16 --cache-type-v f16 -ctkd q8_0 -ctvd q8_0 -devd Vulkan3 --spec-type draft-mtp,ngram-mod --spec-draft-n-max 7 --spec-ngram-mod-n-min 7 --spec-ngram-draft-n-max 31 --spec-rs-depth 24 --fit-verify on
Use this if you have 3 cards, if you have less maybe just remove it -b 8192
In case you want to know how many tokens you have use, --token-ledger /path_to_where_you_want_to_store_stats/token_ledger.json
here is how logs look like I saw a 350t/s on tg but was because you hit ngram mod tables over things the model has already seen
457.04.985.808 I srv print_line: token ledger: requests = 27077, prompt = 3928887769 (cached = 3874707808, processed = 47151296), generated = 22126308 (reasoning = 15129037), t> 457.04.985.913 I slot rq_write_fin: id 0 | task -1 | request timeline: task 77431, ttft 1580.0 ms (http 20.4, tokenize 65.1, queue 247.1, select 0.0, restore 2.7, prompt 1244.3),> 457.05.202.640 I srv handle_compl: task created: conversation_id=64d89c6d7ff76923 (tokens 0..18489, system+first_user) 457.05.202.757 I slot get_availabl: id 0 | task -1 | selected slot by LCP similarity, f_sim_best = 0.994 (> 0.100 thold), f_keep = 1.000 457.05.206.552 I slot launch_slot_: id 0 | task 77458 | processing task, is_child = 0 457.05.207.188 I slot operator(): id 0 | task 77458 | prompt processing batch, tokens 81032..81036 of 81552 457.05.246.405 I slot operator(): id 0 | task 77458 | prompt processing batch, tokens 81036..81548 of 81552 457.05.310.149 I srv lazify_check: checkpoint: converted pos_min=80943 to disk-backed ('cp_64d89c6d7ff76923_80943.bin', 149.6 MiB released) 457.05.872.722 I slot operator(): id 0 | task 77458 | prompt processing batch, tokens 81548..81552 of 81552 457.05.943.921 I srv lazify_check: checkpoint: converted pos_min=81035 to disk-backed ('cp_64d89c6d7ff76923_81035.bin', 149.6 MiB released) 457.09.000.990 I slot print_timing: id 0 | task 77458 | n_gen = 340, tg = 112.88 t/s, tg_3s = 113.20 t/s 457.11.161.223 I slot print_timing: id 0 | task 77458 | prompt eval time = 790.96 ms / 520 tokens ( 1.52 ms per token, 657.43 tokens per second) 457.11.161.226 I slot print_timing: id 0 | task 77458 | eval time = 5163.27 ms / 642 tokens ( 8.06 ms per token, 124.15 tokens per second) 457.11.161.226 I slot print_timing: id 0 | task 77458 | total time = 5954.23 ms / 1162 tokens 457.11.161.227 I slot print_timing: id 0 | task 77458 | graphs reused = 72319 (pp skipped 525) 457.11.161.228 I slot print_timing: id 0 | task 77458 | graphs prebuilt = 7295 (hit 6975, miss 320) 457.11.161.229 I slot print_timing: id 0 | task 77458 | early prebuilt = 28686 (fit 44246, hit 26411, miss 2275, aborted 0, 96584.2 ms host) 457.11.161.238 I slot print_timing: id 0 | task 77458 | draft acceptance = 0.83204 ( 535 accepted / 643 generated), mean len = 6.00 457.11.161.243 I slot print_timing: id 0 | task 77458 | draft auto = n=6 | 1:55.4 t/s [2.00 tok / 36.1 ms] (4) 2:67.7 t/s [2.50 tok / 37.0 ms] (4) 3:101.0 t/s [4.00 tok / 39.6 ms> 457.11.161.251 I slot print_timing: id 0 | task 77458 | draft by impl: draft-mtp | steps 101 | built 5.5 | offered 5.5 | accepted 4.9 (0.90) | full 80 | step 45.2 ms | 131.5 t/s > 457.11.161.257 I slot print_timing: id 0 | task 77458 | draft by impl: ngram-mod | steps 6 | built 64.0 | offered 14.3 | accepted 6.0 (0.42) | full 1 | step 90.8 ms | 77.1 t/s | > 457.11.161.258 I slot print_timing: id 0 | task 77458 | ngram policy: auto on | cap now 12 | cap min 7 median 12 max 31 over 6 chains | cap bound 6 | whole ema 0.17 | resume 0 | > 457.11.161.259 I slot print_timing: id 0 | task 77458 | spec rollback: prebatch restores 1 | host checkpoints 0 | checkpoint restores 0 457.11.161.260 I slot print_timing: id 0 | task 77458 | chain truth: ngram | chains 6 | censored 0 | would land mean 7.0 | full 0 | hist 0:0 1-4:1 5-8:3 9-16:2 17-32:0 33-48:0 49> 457.11.161.261 I slot print_timing: id 0 | task 77458 | ngram gate: held 38 | agree 5 (would land 0:0 1-4:1 5-16:4 17+:0, pending 0) | disagree 33 (would land 0:33 1-4:0 5-16:0 1> 457.11.161.261 I slot print_timing: id 0 | task 77458 | ext shadow: mtp+ngram | attempts 101 | hits 85 | tested 68 | landed mean 0.4 (over tested) | hist 0:65 1-4:1 5-8:0 9-16:2 > 457.11.161.262 I slot print_timing: id 0 | task 77458 | alt table: n=48 | attempts 101 | hits 45 | would land mean 0.0 | hist 0:45 1-4:0 5-8:0 9-16:0 17-32:0 33-48:0 49-64:0 65+:> 457.11.161.263 I slot print_timing: id 0 | task 77458 | mtp ahead: tried 53 (0 after a p_min stop) | verify full 49 | bonus matched 48 (0 after p_min) | too short 0 | served 48 s> 457.11.161.265 I slot print_timing: id 0 | task 77458 | mtp confidence: stop cap:101(acc 4.9) pmin:0(acc 0.0) other:0 | p bins landed/tested: <0.5 1/3 0.5 0/1 0.6 1/1 0.7 0/0 0.8> 457.11.163.234 I slot release: id 0 | task 77458 | stop processing: n_tokens = 82195, truncated = 0 457.11.163.426 I srv print_line: token ledger: requests = 27078, prompt = 3928969321 (cached = 3874788840, processed = 47151816), generated = 22126950 (reasoning = 15129071), t> 457.11.163.526 I slot rq_write_fin: id 0 | task -1 | request timeline: task 77458, ttft 884.3 ms (http 21.2, tokenize 68.3, queue 0.1, select 0.0, restore 3.3, prompt 791.0), gen> 457.11.396.231 I srv handle_compl: task created: conversation_id=64d89c6d7ff76923 (tokens 0..18489, system+first_user)