News for hackers. Stuff that matters.
Re:How Transformers Work by swyx on Thursday October 05, 2023 @11:43AM (#37780084)
> Most models do something called top-p not beam search these days

now im not so sure you know what youre talking about. gpt4 at least is rumored to use beam search. as i understand this is more sophisticated than top-p, so thats noteworthy

Re:How Transformers Work by spmurrayzzz on Thursday October 05, 2023 @12:30PM (#37780682)
While I agree that its an overstatement to say beam search is a minor/worthless detail, its not as cut and dry as "beam search > top-p". You will get wildly variant results with both sampling strategies depending on the objective function that any given model was trained on, in addition to the type of task domain you're expecting the model to perform.

Just to cherry pick one domain: if you're doing any creative work (storytelling, roleplaying, etc), top-p may be a better sampling strategy because you will almost always get more diverse output. But since it's much less predictable, you probably don't want that sort of strategy for summarization or sentiment analysis.