Re:What Are Transformer Models and How Do They Work?
by toxik on Saturday April 15, 2023 @05:41AM
(#35579060)
To expand on this, one of the most common tricks is Nucleus sampling. Roughly, you zero out the lowest probabilities such that the remaining sum to just above some threshold you decide (often around 80%).
The idea is that this is more general than eg changing the temperature of the softmax, or using top-k where you just keep the k most probable outcomes.
Re:What Are Transformer Models and How Do They Work?
by jaidhyani on Sunday April 16, 2023 @12:37PM
(#35591571)
That's true, but they didn't go into any other applications in this explainer and were presenting it strictly as a next-word-predictor. If they are going to include final softmax, they should explain why it's useful. It would be improved by being simpler or more comprehensive, but complexity without reason is bad pedagogy.
Selecting the likeliest token is only one of many sampling options, and it's extremely poor for most tasks, moreso when you consider the relationships between multiple executions of the model. _Some_ (not necessarily softmax) probability renormalization trained into the model is issential for a lot of techniques.