gebdev

I’ve been thinking about this same idea recently, so I’m glad it exists now!

The biggest benefit I see is to enable RAM constrained GPUs to perform inference of large parameter models with surprisingly high throughput. Because only a single expert is resident, the memory to compute ratio over the network is limited only by the activations, not the weights. For an Moe like kimi k3 where active parameters are 103B, we might expect to achieve performance limited only by ~5 effective tok/s per Tflop and ~1 tok/s per 100GB/s.

The more members of the network, the smaller your resident parameters are required to be. I’m not sure whether we can split layer inference into arbitrary chunks, but if so you’d be able to increase memory throughput by storing everything in GPU caches.

Of course, we expect latency to be relatively high, but that’s a tradeoff that's fine for certain circumstances.

I’m not sure whether there are any issues more with this idea, but it’s a fun one nonetheless :)

Loved this article. It showed how lacking my knowledge is in how operating systems implement concurrency primitives. It motivated me to do a bunch of research and learn more.

Notably the claim about how atomic operations clear the cache line in every cpu. Wow! Shared data can really be a performance limitation.

> My experience with arenas is that they increase performance at the cost of a little extra memory usage.

ack, thanks! You’re exactly right, I got my words mixed up.

This article is really well written. I like how it defines a new concept (bughouse chess), then uses it to help describe an emotion they’ve been feeling wrt more popular culture.

I also think bughouse seems cool (aside from the issues mentioned), and want to give it a shot now. Probably in-person.

This is a great example of how ADTs can be implemented in C by emulating classes, despite the loss in brevity.

For the first item on reference counting, batched memory management is a possible alternative that still fits the C style. The use of something like an arena allocator approximates a memory lifetime, which can be a powerful safety tool. When you free the allocator, all pages are freed at once. Not only is this less error prone, but it can decrease performance. There’s no need to allocate and free each reference counted pointer, nor store reference counts, when one can free the entire allocator after argument parsing is done.

This also decreases fallible error handling: The callee doesn’t need to free anything because the allocator is owned by the caller.

Of course, the use of allocators does not make sense in every setting, but for common lifetimes such as: once per frame, the length of a specific algorithm, or even application scope, it’s an awesome tool!

While I tend towards the side of the article, I find it difficult to agree (or follow) many of the points it makes, which is a bit disappointing.

For example, under “ChatGPT Is So Popular”, they disagree with the premise then use the argument that “ChatGPT was marketed with lies” as evidence. The later argument is well researched, but is simply out of place, leaving nothing to support their disagreement.

This is an interesting study. I wonder how the LLM option might compare to human written responses in the same format (but with higher latency), or even to having a physical human in the room. Given some of the points from the conclusion about teachers being able to detect LLM inspired work, I wonder if either of these options may, at times, be better forms of learning due to improved quality.