garrinm

Originally I had those parts written in math with probability functions and the likes (its closer to my background). Then I remembered who is my target audience... but now that I see exactly who is my target audience I'm thinking I'll should have snuck a pelican in there. All jokes aside I appreciate the comment and I'm glad that rewrite paid off!

To be more specific there’s no ground truth tokens to predict. There a verifiable answer in RLVR. But the tokens are explored. Not predicted as there’s no true token to predict.

I try to make 3 claims in the post, it was a bit clumsy I'll admit that.

1. At inference time, LLMs emit one token at a time given the prior tokens. This looks like prediction and I concede that.

2. During pre-training, LLMs predict the next token and compare to the actual next token in the training data. This is the classic setting for ML predictions. And I think its meaningful, the model really is predicting what the ground truth next token will be in the data.

3. During post-training, in the case of RLVR, there is no ground truth next token. In pretraining, the question is "what token actually came next?". In RLVR, the question is "what sequence of actions gets a high reward?"

And the whole point is that thinking about the RLVR is important. A mental model that stops at 1 or 2 is incomplete and doesn't capture what drives LLM tokens.

Yes I understand the analogy was a bit loose. I'm comparing what happens at "inference time" in chess engines to what happens at train time in LLMs. In hindsight AlphaGo Zero was the perfect analogy, but I missed that opportunity.

The analogy with chess still works, but there's an extra step to think about. In both cases there is some kind of search over possible future trajectories. A chess engine explicitly searches branches of the game tree and evaluates which moves lead to good outcomes. In RL for an LLM, you sample rollouts, evaluate the resulting trajectories, and use those evaluations to update the policy.

The extra step with the LLM is that you don't keep doing that whole search at inference time. You use the rollouts to update the weights, so in some sense the useful information from that search gets compressed into the model.

But if you accept that the model is, in some loose sense, storing what it learned from those rollouts in its weights, then at inference time they are doing a similar job: taking some input state (prior tokens or a board position) and choosing the next action.

In the article I made 3 claims, and I agree it was a bit clumsy.

1st I say that "working forwards" in the sense of outputting one token at a time could be some form of prediction, I don't argue against that. This is what LLMs do at inference time.

2nd I say that to me what really constitutes a prediction is the pre-training. Here it's the classic setting for the word prediction in ML. The model outputs a prediction of the ground truth label: the next token.

3rd I argue that in RL there is no ground truth next token, so prediction doesn't apply here anymore.

Back to your question then: you're asking points 3 and 1 are different. Working backwards from a set of win states is basically what RL does in training. Working forward from the current state is what inference does. To me there is a distinction worth thinking about. First between the mechanism at inference time and at train time. Then between what happens in pre-training vs. RL post training.

I think the point is more that in RL there's no ground truth to predict. So when training a model with RL the idea of "predicting" doesn't fit anymore. I'll make some edits I see that I wasn't very clear.

It does in pre training, but not in RL post training. And not at inference time. Reading over all these comments I get the feeling my mistake was not clearly delineating inference time and train time.

I think that’s fair, I didn’t actually run the whole thing through an AI. it was more targeted edits, but each time it does erode at my writing. But at the same time, I don’t think it’s a good reason to dismiss this. Because I did spend several hours writing it, and I did put a lot of thought into it, and it was not in any meaningful way generated by AI.

The distinction I perhaps didn’t make clearly enough is that I’m not really debating the concept of prediction at inference time, although, as I pointed out elsewhere, I think that’s the less interesting interpretation of what “prediction” means.

What’s more interesting to me is its application at training time. In reinforcement learning, there is no ground-truth next token to predict.

So if you’re comfortable calling Deep Blue a “next move predictor,” then I think it’s perfectly consistent to call an LLM a “next token predictor.” But I think it’s more useful to think of Deep Blue as evaluating the value of possible moves. roughly, how likely they are to lead to winning.

And I think effectively the same distinction applies here.

It was written by a human. There are AI edits but it’s very much a human composition. Perhaps a bit sloppy.

Yes, I think that’s a good explanation. There are really two sides to it.

There’s the mechanical, inference time, autoregressive, one-token-after-another side, which I’m not going to argue isn’t prediction. I just think that’s a relatively uninteresting use of the word “prediction,” because it’s effectively a system predicting its own output.

The more interesting question is what happens at training time. As you describe, reinforcement learning allows the model to learn to output things that it never could have learned simply by predicting what appears in the training corpus.

More concretely, in reinforcement learning there are no ground-truth next tokens to predict.

In supervised machine learning, “prediction” usually means there is some ground-truth label that will eventually be revealed. The model predicts what that label is, the difference between the prediction and the truth gives you a loss, and you learn from that.

But in reinforcement learning, there is no ground-truth action waiting to be revealed. The model chooses an action, observes the consequences, and learns from the reward. To me, that’s a meaningfully different thing from prediction.

Thanks for the insight, I wasn't aware of `track_caller`. I'll definitely be looking into this. I was scratching my head trying to figure out how to make file and line number usage consistent and customizable, this looks like the answer!

You're also right that this will pretty much eliminate the need for macros.

That's also a very key insight about Display vs. Debug printing. I'll be looking into that as well.

Thank you for the thoughtful reply.

Anyhow still makes things easier for application development. The main drawback is that the resulting error type doesn't implement std::error::Error, so it's not suitable for library development (as pointed out in the anyhow documentation). Stack Error is a bit less ergonomic, but suitable for library development.

I played around a bit with SNAFU a couple of years ago, but I'm haven't worked deeply with the library so there might well be some features I'm not aware of.

I think SNAFU is more like a combination of anyhow and thiserror into a single crate, rather than Stack Error which leans more heavily into the "turnkey" error struct. Using the Whatever struct, you get some overlap with Stack Error features:

- Error message are co-located.

- Error type implement std::error::Error (suitable for library development).

- External errors can be wrapped and context can easily be added.

Where Stack Error differs:

- Error codes (and URIs) offer ability for runtime error handling without having to compare strings.

- Provides pseudo-stack by stacking messages.

Underlying this is an opinion I baked into Stack Error: error messages are for debugging, not for runtime error handling. Otherwise all your error strings effectively become part of your public interface since a downstream library can rely on them for error handling.

Stack Error is a pragmatic error handling library for Rust that provides helpful messages for debugging, and structured data for runtime error handling.

Features:

- Informative error messages: stack error messages and optionally add file/line context to your messages. This helps convey not just what went wrong, but also how it went wrong, making debugging faster.

- Programmatic error handling: include optional error codes and URIs for robust runtime handling.

- Library-Friendly: define custom error types easily while staying compatible with Rust’s error ecosystem.

If designing good error structures for your projects slows you down, and you need something more library-friendly and structured than anyhow, Stack Error might be what you’re looking for.

This gets discussed in the Roc community. They are exploring designing a language without higher kinded polymorphism.

Here's a snippet from the Roc FAQ.

> It's impossible for a programming language to be neutral on this. If the language doesn't support HKP, nobody can implement a Monad typeclass (or equivalent) in any way that can be expected to catch on. Advocacy to add HKP to the language will inevitably follow. If the language does support HKP, one or more alternate standard libraries built around monads will inevitably follow, along with corresponding cultural changes. (See Scala for example.) Culturally, to support HKP is to take a side, and to decline to support it is also to take a side.

https://www.roc-lang.org/faq.html#higher-kinded-polymorphism

Yes, Clint is a proof-of-concept, meant to showcase this use case of LLMs more than anything else.

Unfortunately you can ask Clint to tell you just about anything. But fortunately it will at least try to tell you that some things are less plausible than others.

Clint should be used only to research information. It provides links to resources. It uses Retrieval Augmented Generation which is less prone providing incorrect information, though it can still happen.

Clint should not be used for diagnosis. Only for personal information. Like an web search but more interactive.