...especially with megakernels...

I don't understand why megakernels haven't become the default choice. The speed advantage is crazy. I made a few changes to the openjet megakernel to run in a dual gpu (older gen, not blackwell) box and doubled my token generation speed on qwen3.8-27B (4-bit though).

I keep debating adding a third matching gpu to see if I can push the speed a bit more and move back to the 8-bit quant, but I'm stuck on the idea of just building a Blackwell gpu system despite the cost factor. Plus a third gpu is going to going to be weird for the tensor split and P2P data syncs and my workstation is old with 3rd gen PCIe etc etc.

Architectures (like Gemma4) have things like sliding window attention that make it pretty impractical to do on a GPU, you would have to do additional things to make it work, and the talent to do that, even with LLM assistance is very small.

Ah, that makes sense. I haven't bothered to look at current low-level NN architecture details. I'm a little surprised that there isn't a bumper crop of relatively well-informed talent to do gpgpu work. Although, when if I think about it, it has always been a little bit of a niche area.

The context window left on my 24GB card with Gemma is to low to do any major task though. Maybe small diffs or a single direct task. But Qwen has enough context for agentic tasks.

32k context on my 3090 runs fine.

32k context is quite limited in agentic context. Qwen 3.8 27b is much better in agentic and coding workflows. Gemma4 31b is much better in general assistant and language focused workflows.

Realistically a 32GB card is what you need to run Gemma4:31b well. Doing it in 24GB leaves almost no room for context / KV cache.

32k context is plenty fine. thats 50 pages of single spaced text

Not if you preserve reasoning in the KV cache, 32k ctx is nothing.