kcorbitt

And lately, the sweet spot has been moving upwards every 6-8 weeks with the model release cycle.

Is it?

Dang, hadn't seen that. Namespace collision strikes again.

I really like RLPR for when you have a known-good answer to compare to as well!

No, we don't do anything. Theoretically we could judge several times with different ordering.

We could measure order bias really easily though; we just need to look at the average score by rollout position across many runs. I'll add that to my list of experiments!

Thank! If there are any topics that you'd find particularly interesting, let me know and I can try to find time. :)

Looks cool! With vLLM v1, prefix caching is enabled by default and seems quite performant. Is the advantage of LMCache the fact that you can offload to CPU and disk as well? How much is throughput/latency affected if you need to pull a large KV cache from disk/cpu instead of GPU RAM?

Also, how realistic would it be to share the KV cache across vllm nodes within a data center? It would be really nice to be able to freely distribute requests to a pool of vLLM workers without worrying about prefix-aware routing, but maybe that isn't the right approach because moving the KV cache around would be too slow?

I was curious about this so I had o3 do a bit of research. Turns out 300 L40s have more compute than any supercomputer before 2013 (and arguably before 2016, depending on how you count reduced-precision FLOPs).

https://chatgpt.com/share/685dea79-26ec-8002-bd62-7ed83aedf4...

The real answer is that nobody trusts their automated evals enough to be confident that any given automatically-trained release actually improves performance, even if eval scores go up. So for now everyone batches up updates and vibe-checks them before rolling them out.

It seems like the speedups here are most useful for small models, since on larger models a smaller fraction of the total time would be spent swapping between kernels? Would be interesting to see at least theoretical results for LLMs in the 14-70B parameter range, which is what most folks deploy in practice.

And of course the effect on throughput at larger batch sizes, which they allude to at the end.

Overall a very interesting result!

There are many industries where you need lots of experience before you're a net contributor to productivity. This is true for everything from hairdressers to doctors. We have ways of dealing with this (eg. taking out loans to undergo years of training).

The problem comes if the number of years of experience you need to outperform the frontier AI models advances at more than 1 per year, which is not out of the question.

I wonder if they've trained the model to operate with a shallower stack; eg. the full model may be composed of 24 transformer blocks, but they've also trained it to accept embeddings at layer 8, so it can be operated with just 16 transformer blocks on lower-resourced devices.

Experimenters in the open source tinkering community have done the opposite (copy/pasting layers in existing models to make them deeper) and it seems to work... fine, with minimal post-training on the new, deeper model required to exceed the performance of the original model. So it's not a crazy idea.

It's very unlikely that they're doing their own pre-training, which is the longest and most expensive part of creating a frontier model (if they were, they'd likely brag about it).

Most likely they built this as a post-train of an open model that is already strong on coding like Qwen 2.5.

For "that last 10% of reliability" RL is actually working pretty well right now too! https://openpipe.ai/blog/art-e-mail-agent

Ok good questions here.

By fine-tuning in this context I assume you mean "supervised fine-tuning", or SFT. SFT trains a model to produce a specific string of output tokens, given an input. With SFT, if you were trying to train an assistant to solve math problems using a code interpreter, you might train it on a dataset that looks like:

    input: 'What is 934+1208'  
    output: `print(934+1208)`

    input: 'how many "r"s in strawberry'
    output: `print(len([l for l in "strawberry" if l == 'r'])`

etc, etc.

RL, on the other hand, just means training a model not to produce a concrete string of output tokens, but rather to create an output that maximizes some reward function (you get to decide on the reward).

For the example above, you might create the following dataset for RL training:

    input: 'What is 934+1208'
    ground_truth: 2142

    input: 'how many "r"s in strawberry'
    ground_truth: 3

You would then train the model to write python code that produces the ground_truth output. Your training code would take the model's output, run the python it produced, and then check whether the output matches the expected ground_truth. Importantly, this doesn't require you actually writing the code to solve the problem (you don't even have to know if it's solvable, technically!). Over time, the training loop would make the model more likely to produce outputs that get high rewards, which hopefully means it gets better at producing valid and applicable python.

This is useful in lots of domains where it's easier to check the answer than actually produce it. In the blog post[1] linked above, we train the agent to effectively use keyword search to try to find the correct emails in an inbox. As the model trainer, I didn't actually know what the right strategy was to choose keywords that would most quickly find the relevant email, but through training with RL, the model was able to figure it out on its own!

[1]: https://openpipe.ai/blog/art-e-mail-agent?refresh=1746030513...

Figured now was a good time to post this since we recently got surprisingly good results on training an email research agent. Link is above, but will put it here as well since I think it's a good example of RL's promise: https://openpipe.ai/blog/art-e-mail-agent

We may be in a simulation, but your odds of being alive to see this (conditioned on being born as a human at some point) aren't that low. Around 7% of all humans ever born are alive today!

Yep. And tbh you probably don't even have to do this; the R1 paper found that just running SFT the base model with a relatively small number of monolingual reasoning traces was enough for it to get the idea and iirc they didn't even bother selecting for language specifically in the RL training looop itself.

To be honest, I don't expect the performance to generalize to other task types with this specific training regime. If we had a panel of like 30 logic puzzles and cross-trained against all of them simultaneously it might though.

I think there's a lot of benefit to discovering a training regime that allows small specialized models to do extremely well in one narrow task; if we can figure out how to make small models that beat SOTA on a specific task and are cheap to train and run, that's in some ways a more useful outcome than a very large model that is good at many tasks (but is more expensive to run for each of them).

One of the authors here. Happy to answer any questions about our methods/results!

OpenPipe | ML & Full-Stack Engineers | Full-time | Seattle, WA (ONSITE) | https://openpipe.ai/ | Highly competitive pay + equity We've built the world's best fine-tuning platform in just over a year. First to launch self-service preference tuning, integrated evals/data prep/fine-tuning, and self-service learning from human feedback.

Our customers range from fast-growing startups to Fortune 500s. We're growing 40% MoM and have achieved this with a team of just 5 engineers, including the founders.

Seeking:

- ML Engineers: Help advance our fine-tuning capabilities. (We have thousands of datasets and evals; testing new ideas is really easy!)

- Systems Engineers: Scale our platform serving hundreds of millions of daily requests

- Full-stack Engineers: Build end-to-end features in TypeScript/Python

Ideal candidates:

- Strong programming skills (TypeScript/Python)

- Systems architecture expertise

- Self-starters (founder/founding engineer experience valued)

- Willing to relocate to Seattle

Highly competitive salary + equity. Well-funded with large customers. Perfect for future founders wanting startup experience or engineers breaking into AI/ML with immediate impact.

Email: [email protected] Include something impressive you've built.

Lots of folks working on open-source reasoning models trained with reinforcement learning right now. The best one atm appears to be Alibaba's 32B-parameter QwQ: https://qwenlm.github.io/blog/qwq-32b-preview/

I also recently wrote a blog explaining how reinforcement fine-tuning works, which is likely at least part of the pipeline used to train o1: https://openpipe.ai/blog/openai-rft

OpenPipe | ML & Full-Stack Engineers | Full-time | Seattle, WA (ONSITE) | https://openpipe.ai/ | Highly competitive pay + equity

We've built the world's best fine-tuning platform in just over a year. First to launch self-service preference tuning, integrated evals/data prep/fine-tuning, and self-service learning from human feedback.

Our customers range from fast-growing startups to Fortune 500s. We're growing 40% MoM and have achieved this with a team of just 5 engineers, including the founders.

Seeking:

- ML Engineers: Help advance our fine-tuning capabilities. (We have thousands of datasets and evals; testing new ideas is really easy!)

- Systems Engineers: Scale our platform serving hundreds of millions of daily requests

- Full-stack Engineers: Build end-to-end features in TypeScript/Python

Ideal candidates:

- Strong programming skills (TypeScript/Python)

- Systems architecture expertise

- Self-starters (founder/founding engineer experience valued)

- Willing to relocate to Seattle

Highly competitive salary + equity. Well-funded with large customers. Perfect for future founders wanting startup experience or engineers breaking into AI/ML with immediate impact.

Email: [email protected] Include something impressive you've built.