Re:Yann LeCun: ChatGPT is 'not particularly innovative'
by smoldesu on Thursday January 26, 2023 @02:19PM
(#34537216)
Heuristics is the ability to recognize multiple options and reason which one is correct. ChatGPT (and all GPT models for that matter) have a single context that doesn't get compared to other possible generations. You can back-propegate and see which tokens influenced the output most, but there is no evidence that AI can conceive multiple hypothetical responses and select the most-preferred one. At least not ChatGPT.
Re:Yann LeCun: ChatGPT is 'not particularly innovative'
by visarga on Thursday January 26, 2023 @03:01PM
(#34537901)
Have you heard of "beam search"? The model keeps a number of optional lines of text without being greedy and picking only the most probable token every time. You can also sample with T>0 multiple times to get an ensemble of answers that can be used as input for another AI operation. You just need to do multiple interactions with the model.
Iām not sure these are well-defined enough terms to make this claim.