charles_irl

Thanks!

We chose the Qwen model for the initial and core results because it's a simple architecture, which made the roofline cost modeling very easy and trustworthy. I'd already done related work with that model before for another project on speculative decoding, so we had a reference impl as well.

For brevity, this post focuses on a subset of the results. In the companion post on the Full Stack Data Lab blog[1], we go into more detail. With any luck, that'll be a nice VLDB-bound arXiV paper soon!

We have some more results on DiffusionGemma that we tacked on after Jev started popping off. Hard to say much about the specific Jev model, since the details are not known -- even to me, working with their team as they scale up on our platform.

But generally, I think a Quail-like query planner-based approach is a slam dunk for this broad set of classifier models. Quite likely that the benefit is primarily for backend/analytics queries though, where the superior KV management has a chance to shine.

2B tokens/minute/H100 happens to be roughly the speed of light for Qwen 4B. Probably unattainable, for the reasons we indicated (and others, e.g. manufacturer overstatement of peak rates), but I wouldn't be surprised if we could squeeze another 20-30% with CUDA PDL or a megakernel.

[1] https://fsdatalab.github.io/blog/introducing-quail/

Very cool!

> Unless you've actually watched tokens stream at those rates, the numbers are hard to internalize. This is the rendering.

I built something similar recently, for the same reason: https://modal.com/llm-almanac/token-timing-simulator.

I like that the output rendering is closer to typical UIs -- syntax highlighting in code mode, tool calls, dim-italic reasoning.

One feature mine has that the author, or anyone else who vibe codes their own version after seeing this, might like to steal is modeling the distribution of output latencies. My implementation is hacky (log-normal roughyl estimated from p50, p90, and p99 values), but still, when you set those to realistic values, it recreates the "jitter" you see in many LLM UIs.

antirez is right that generation tok/s isn't flat as a function of context length, which is a weakness of both simulators.

To clarify: we do content-based hashing, and when we say "shared bytes aren’t guaranteed to be in the exact same container image layer", what we mean is that

FROM some/image RUN pip install torch==2.7.1

and

FROM another/image RUN pip install torch==2.7.1

will produce images with very high overlap in contents, which will be shared by a content-based cache, but those images' final layers are disjoint from the perspective of a layerwise cache.

You're absolutely right!

Yep! That should start in ten seconds or so -- about a second per gigabyte of weights, plus a second to start the container and a few seconds to load the memory snapshot.

There are a few limitations with snapshotting, e.g. it generally fails when using multiple GPUs, which we document here: https://modal.com/docs/guide/memory-snapshots.

Cutting latencies by 40x! Unfortunately couldn't fit the whole title in the character limit :<

> Ed Zitron also called out the business model of GPU-as-a-service middleman companies like modal deeply unsustainable, and I also don't see how they can make a profit if they are only reselling public clouds.

You got a link for that? I work on Modal and would be interested in seeing the argument!

We think building a proper software layer for multitenant demand aggregation on top of the public clouds is sufficient value-add to be a sustainable business (cf DBRX and Snowflake).

Sorry to lead with a bunch of jargon! Wanted to make it obvious that we'd give concrete recommendations instead of palaver.

The technical terms there are later explained and diagrammed, and the recommendations derived from something close to first principles (e.g. roofline analysis).

oof ty, willfix

Inspired by https://float.exposed, which was on the front page recently, I put together this visualizer for lower precision/quantized floating point numbers -- specifically, all of the formats in the Open Compute Project's Microscaling Formats Spec (https://www.opencompute.org/documents/ocp-microscaling-forma...).

Thanks! I think computers are fun and I want reading about them to be fun too.

I was also reminded of HazyResearch's MegaKernels. Didn't want to distract from the main thrust of the post, but definitely think that's a promising approach.

Hey, one of the authors here!

Reductively, software engineering means taking an idea and mapping it into code. So one form of "reverse" engineering would be taking the code and extracting the ideas. That's what we did here.

Because the source is public, there's quite a lot to work with from the start -- the warp specializations are named and there are helpful comments in many places.

But for many components, we didn't have much. Maybe the clearest case of "reverse engineering" explained in the post is with the cubic approximation for the rational part of the exponentiation. That required staring at some inline assembly and doing math.