tjbai

I agree that there's an exploration-exploitation tradeoff, but for what you specifically suggest wouldn't you presumably just normalize by sample size? You wouldn't allocate based off total conversions, but rather a percentage.

From a purely technical definition of bias (difference in expected value of the estimator and the true value), MAB is not biased because "changing the experiment parameters" is just dynamically allocating a different sample size to each of the estimators, so the estimator still converges to the correct value.

You are correct that this setup can potentially mislead you, but this is because you might end up getting estimators with high variance. So, you might mistakenly see some early promising results for experiment group A and greedily assign all the requests to that group, even though it is not guaranteed that A is actually better than B.

This is the famous exploration-exploitation dilemma—should you maximize conversions by diverting everyone to group A or still try to collect more data from group B?

The last hidden state is just the output embedding after N residual layers, e.g. input embedding + res1 + res2 + ...

There's typically an "unembedding layer"/"classification head" that uses this hidden state to produce a softmax distribution over the LLM's vocabulary. In this case, we can think of this as "snapping" the hidden state into a single token and feeding that token into the next position of the autoregressive LLM.

In this sense, the last hidden state _does_ augment the next input. The authors simply propose directly feeding this hidden state into the next step rather than reducing it into a single token—thus, reasoning in continuous latent space rather than discrete token space.

Something can be inconsequential and yet still interesting/amusing

> The multi-armed bandit is a really interesting problem in mathematical optimization... This problem is so interesting, in fact, that during World War II the Allies proposed air dropping copies of the original paper over Germany. The end result, or so it was theorized, was that German scientists would be so fascinated and distracted by the problem that they would abandon the war effort and cripple any German military research projects in the process.

This is incredible