News for hackers. Stuff that matters.
Re:Inkling 2 Mini by xlayn on Friday November 20, 2026 @04:55PM (#50582827)
Hey, just in case.. I have been maintaining this fork of llama.cpp, I run also 7900XTX, 3 in my case... and I want to know if you see improved speed... notice the ngram mod where instead of loosing the ngrams they are stored in disk, I have a 31M slot and so far it's being working really nice for me

I'm seing around 2800t/s on pp at the very beginning of the conversation, and then going to around 800 by the 256k context size...

I've also update the value in the gguf to have instead of context size 256k be triple and have use it all the way to around 600k context size

use the cache eviction branch https://github.com/alainnothere/llama.cpp/tree/disk-cache-ev...

llama-server -m /path_to_model/Qwen3.8-27B-UD-IQ4XS_u.gguf -np 2 --host 0.0.0.0 --port 8089 --kv-unified --temp 0.2 --top-k 20 --min-p 0.0 --cache-disk-path /path_to_store_conversations_to_disk --cache-ram 100 --cpu-range 0-23 --reasoning-budget 4096 --reasoning-budget-message "... I am thinking for too long -- let me gather more info about the task." --no-cache-idle-slots --spec-draft-auto --spec-draft-sampling probabilistic --spec-draft-n-min 1 --spec-draft-p-min 0 --spec-ngram-auto --spec-ngram-mod-size 35544320 --spec-ngram-mod-cache /path_to_storage_of_ngram_mod_this_is_specific_per_model/ngram-mod-qwen3.8-27b.bin --fit on --device Vulkan3,Vulkan2,Vulkan1 --fit-target 1024,2560,2560 --ctx-checkpoints 25 --checkpoint-spill-max 25 --cache-type-k f16 --cache-type-v f16 -ctkd q8_0 -ctvd q8_0 -devd Vulkan3 --spec-type draft-mtp,ngram-mod --spec-draft-n-max 7 --spec-ngram-mod-n-min 7 --spec-ngram-draft-n-max 31 --spec-rs-depth 24 --fit-verify on

Use this if you have 3 cards, if you have less maybe just remove it -b 8192

In case you want to know how many tokens you have use, --token-ledger /path_to_where_you_want_to_store_stats/token_ledger.json

here is how logs look like I saw a 350t/s on tg but was because you hit ngram mod tables over things the model has already seen

457.04.985.808 I srv print_line: token ledger: requests = 27077, prompt = 3928887769 (cached = 3874707808, processed = 47151296), generated = 22126308 (reasoning = 15129037), t> 457.04.985.913 I slot rq_write_fin: id 0 | task -1 | request timeline: task 77431, ttft 1580.0 ms (http 20.4, tokenize 65.1, queue 247.1, select 0.0, restore 2.7, prompt 1244.3),> 457.05.202.640 I srv handle_compl: task created: conversation_id=64d89c6d7ff76923 (tokens 0..18489, system+first_user) 457.05.202.757 I slot get_availabl: id 0 | task -1 | selected slot by LCP similarity, f_sim_best = 0.994 (> 0.100 thold), f_keep = 1.000 457.05.206.552 I slot launch_slot_: id 0 | task 77458 | processing task, is_child = 0 457.05.207.188 I slot operator(): id 0 | task 77458 | prompt processing batch, tokens 81032..81036 of 81552 457.05.246.405 I slot operator(): id 0 | task 77458 | prompt processing batch, tokens 81036..81548 of 81552 457.05.310.149 I srv lazify_check: checkpoint: converted pos_min=80943 to disk-backed ('cp_64d89c6d7ff76923_80943.bin', 149.6 MiB released) 457.05.872.722 I slot operator(): id 0 | task 77458 | prompt processing batch, tokens 81548..81552 of 81552 457.05.943.921 I srv lazify_check: checkpoint: converted pos_min=81035 to disk-backed ('cp_64d89c6d7ff76923_81035.bin', 149.6 MiB released) 457.09.000.990 I slot print_timing: id 0 | task 77458 | n_gen = 340, tg = 112.88 t/s, tg_3s = 113.20 t/s 457.11.161.223 I slot print_timing: id 0 | task 77458 | prompt eval time = 790.96 ms / 520 tokens ( 1.52 ms per token, 657.43 tokens per second) 457.11.161.226 I slot print_timing: id 0 | task 77458 | eval time = 5163.27 ms / 642 tokens ( 8.06 ms per token, 124.15 tokens per second) 457.11.161.226 I slot print_timing: id 0 | task 77458 | total time = 5954.23 ms / 1162 tokens 457.11.161.227 I slot print_timing: id 0 | task 77458 | graphs reused = 72319 (pp skipped 525) 457.11.161.228 I slot print_timing: id 0 | task 77458 | graphs prebuilt = 7295 (hit 6975, miss 320) 457.11.161.229 I slot print_timing: id 0 | task 77458 | early prebuilt = 28686 (fit 44246, hit 26411, miss 2275, aborted 0, 96584.2 ms host) 457.11.161.238 I slot print_timing: id 0 | task 77458 | draft acceptance = 0.83204 ( 535 accepted / 643 generated), mean len = 6.00 457.11.161.243 I slot print_timing: id 0 | task 77458 | draft auto = n=6 | 1:55.4 t/s [2.00 tok / 36.1 ms] (4) 2:67.7 t/s [2.50 tok / 37.0 ms] (4) 3:101.0 t/s [4.00 tok / 39.6 ms> 457.11.161.251 I slot print_timing: id 0 | task 77458 | draft by impl: draft-mtp | steps 101 | built 5.5 | offered 5.5 | accepted 4.9 (0.90) | full 80 | step 45.2 ms | 131.5 t/s > 457.11.161.257 I slot print_timing: id 0 | task 77458 | draft by impl: ngram-mod | steps 6 | built 64.0 | offered 14.3 | accepted 6.0 (0.42) | full 1 | step 90.8 ms | 77.1 t/s | > 457.11.161.258 I slot print_timing: id 0 | task 77458 | ngram policy: auto on | cap now 12 | cap min 7 median 12 max 31 over 6 chains | cap bound 6 | whole ema 0.17 | resume 0 | > 457.11.161.259 I slot print_timing: id 0 | task 77458 | spec rollback: prebatch restores 1 | host checkpoints 0 | checkpoint restores 0 457.11.161.260 I slot print_timing: id 0 | task 77458 | chain truth: ngram | chains 6 | censored 0 | would land mean 7.0 | full 0 | hist 0:0 1-4:1 5-8:3 9-16:2 17-32:0 33-48:0 49> 457.11.161.261 I slot print_timing: id 0 | task 77458 | ngram gate: held 38 | agree 5 (would land 0:0 1-4:1 5-16:4 17+:0, pending 0) | disagree 33 (would land 0:33 1-4:0 5-16:0 1> 457.11.161.261 I slot print_timing: id 0 | task 77458 | ext shadow: mtp+ngram | attempts 101 | hits 85 | tested 68 | landed mean 0.4 (over tested) | hist 0:65 1-4:1 5-8:0 9-16:2 > 457.11.161.262 I slot print_timing: id 0 | task 77458 | alt table: n=48 | attempts 101 | hits 45 | would land mean 0.0 | hist 0:45 1-4:0 5-8:0 9-16:0 17-32:0 33-48:0 49-64:0 65+:> 457.11.161.263 I slot print_timing: id 0 | task 77458 | mtp ahead: tried 53 (0 after a p_min stop) | verify full 49 | bonus matched 48 (0 after p_min) | too short 0 | served 48 s> 457.11.161.265 I slot print_timing: id 0 | task 77458 | mtp confidence: stop cap:101(acc 4.9) pmin:0(acc 0.0) other:0 | p bins landed/tested: <0.5 1/3 0.5 0/1 0.6 1/1 0.7 0/0 0.8> 457.11.163.234 I slot release: id 0 | task 77458 | stop processing: n_tokens = 82195, truncated = 0 457.11.163.426 I srv print_line: token ledger: requests = 27078, prompt = 3928969321 (cached = 3874788840, processed = 47151816), generated = 22126950 (reasoning = 15129071), t> 457.11.163.526 I slot rq_write_fin: id 0 | task -1 | request timeline: task 77458, ttft 884.3 ms (http 21.2, tokenize 68.3, queue 0.1, select 0.0, restore 3.3, prompt 791.0), gen> 457.11.396.231 I srv handle_compl: task created: conversation_id=64d89c6d7ff76923 (tokens 0..18489, system+first_user)

There are no comments on this comment yet.