jakswa

did they stick with the image-based compaction (when vision is supported i guess)? It boggled my mind: "snapcompact, which archives older history into dense images for a vision-capable model"

llama.cpp looks to be cloning openai's realtime websocket API with this audio model as its first use case? I guess this model is either the first speech-to-speech of its kind, OR it's the first good-enough one. I'm thinking I need to give this more testing!

  - https://github.com/ggml-org/llama.cpp/pull/31287
  - https://github.com/ggml-org/llama.cpp/pull/31269

I'm testing this out as a dumb scout model alongside Qwen 3.8 Flash Next (IQ3_S, strata). Currently have it labeled as a "dumb but fast scout, findings should be verified" since some initial testing shows useful-but-sometimes-inaccurate.

on a 7900XT (20GB VRAM, ~800GB/s IO) w/ full context:

    - 8500 tok/s prefill/ingestion
    - 212 toks/s decode/generation

52B active parameters and my qwen-3.8-flash-next model running at home is supposedly competitive? maybe I gotta test it to believe it. https://artificialanalysis.ai/models/comparisons/mistral-lar...

> so many models seem to be at almost exactly 74%

There was talk of deepswe tests being flawed. I have yet to find a better gauge of useful work for my case though, so I still keep referencing it and sampling it. On the lookout for improved benchmarks.

"We found issues in at least 23 out of 113 tasks (over 20.3%), leading us to designate the benchmark as Flawed." - https://epoch.ai/benchmarks/deepswe/review

no apology needed, I didn't find any corrections to suggest in your comment, lgtm

I think you were meaning to reply to another comment?

I was recently scanning for newer, more-batteries-included options and this looks well-scoped at first bluff. Thanks for spreading it, might end up in my next experiment!

If you can splurge for the 40GB+ memory it needs, this genuinely seems like a usability upgrade to 27B while meeting/exceeding capability. I was able to get this to complete each of the 3 easy deepswe exercises I threw at it (on q2!). 27B really struggles on deepswe on my 7900XTX GPU -- thinks too much, and slows down a good chunk as context builds. More usage needed!

While I appreciate the viewpoint, I do not think the idea resonates personally. If I'm following, your idea is to take a brain submerged in this feedback loop from childhood and task it with the question: "what should you want to do?" Are you expecting new results? Is there some noble, obvious result you held back that you think this brain can come up with? Pick ourselves up by our bootstraps to go where?

I think the population starting to age is an interesting one -- the first one playing more-complex 2D + 3D games from an early age. I'm in this age group. In a world with nothing left to explore, and humans globally connected by commercialism, I think you don't need to be bothered with what capitalism is making of the world. Our brains are hooked on solving imaginary, complex problems. We're all on some weird ship at sea, migrating toward some unknown old-age destination. Probably an MMO.

Bonsai 2 27B · Radeon RX 7900 XTX

  - 89 tokens/sec generation with speculative decoding
  - 81 tokens/sec at 20k context
  - 474 tokens/sec ingestion at 20k — about 42 seconds
  - 10.1 GiB peak VRAM with a 24k context window

ROCm 7.2.3 · PQ2_0 · Qwen Q4 MTP, draft length 2

---- versus ----

Qwen3.8-27B IQ3_S · Radeon RX 7900 XTX

  - 79 tokens/sec generation on a short coding prompt
  - 61 tokens/sec at 60k context
  - 53 tokens/sec at 95k context
  - 558 tokens/sec ingestion at 60k — about 108 seconds
  - 19.9 GiB peak VRAM during coding tests with a 100k context window

Vulkan · GSQ-RCO IQ3_S · MTP, draft length 2 · vision projector loaded

what kinda speeds do you see on 6700 XT? i'm always conflicted on investing time chasing speed-vs-quality tradeoffs. I've got a 7900 XT (about double the IO throughput). I'll probably end up giving it a go when I find time.

I love their web UI so much that I had AI slop all over it in a fit of fanboy-ism https://inkcap.click

dang only for certain nvidia GPUs, had my hopes up

don't see it in my AWS bedrock model list yet, but boy has bedrock mantle been annoying today with the errors/downtimes, with NO status page entries >_<

Oh. The UI screenshot on github is... not actually in the github repo? It's platform/hosted only? There's my first awkward discovery, but makes sense in retrospect.

I think I have to look at setting up one of these AI gateways for work, since AWS bedrock is such a PITA to hook a harness up to over IAM roles. Also I still can't believe bedrock hasn't released any open models in months (so there's paranoia that I'll want to swap in another provider).

Really though I'm hoping anthropic fixes the oppressive claude verbosity. I saw someone refer to being "clauderboarded" and my brain cannot let go of this as Claude's tokens bombard me.

ended up disabling ornith 9B. Oddly Ling 3 Tiny is pretty dang capable if its thinking is unleashed (tons of output tokens, maybe 5X the tokens but it's so fast it's maybe only twice as slow as a smarter model). This is a really interesting space if these super small models keep improving. SO FAST! Want to use!

I had to go down to UD-Q3_K_XL for Qwen 3.8 27B to get it to fit in VRAM and be usable, but I worry I'm gutting its intelligence somewhat. I too am interested in faster + more-usable alternative that can exchange blows with the Q3-dumbed 27B.

I'll be comparing the 9B vs Ling 3 Tiny (8B-A1B) as a scout model. Ling tiny is so fast but can be a little too dumb. Hope the 9B strikes a good middleground even if dense/slower.

I've been waiting on this model to show up on the Deep SWE benchmark results and treat its absence/delay as an indication of how slow and unusable it is for good results. I bet it thinks to the moon on some of those complex challenges.

Thanks for mentioning Ling 3 Tiny. This model has completely bypassed me and seems promising for how small it is.

I'm waiting for this comparison too. I was impressed by a 1-shot GLM 5.3 did for me the other day.

Gemma 4 (both E4B + 12B) performed really well as ears+brains. I mostly comment because I too am always scouting for a nice local all-in-one model.

I went back to Glimmer 30b for my 20GB of VRAM. Just a better experience fit-wise and speed-wise and tone-/voice-wise.

thank you I'm still on ie8

I'm in the exact same boat with a 7900 XT and a good Glimmer 30B experience. I was really hoping qwen 3.8 would bring some memory/space efficiency savings along the lines of whatever is going on with Glimmer 30B. I have been surprised that a 30 billion model fits and runs better (at higher unsloth quantization! UD-Q4_K_XL fits!) than a 27 billion model.

anecdotes: 35B-A3B does want more memory, bigger model. But if you get it running it will be faster and more enjoyable to use -- text will fly by -- due to only 3B params being active, in my experience at least.

I dunno, I didn't read in-depth. Hopefully you don't gotta zoom in with human eyeballs.

OMP changed the default compaction to images! Kinda nuts to read about. Saves the generation cost of the traditional compaction step and writes the context as tiny text to an image, if I was following correctly.