News for hackers. Stuff that matters.
Re:Yann LeCun: ChatGPT is 'not particularly innovative' by smoldesu on Thursday January 26, 2023 @02:19PM (#34537216)
Heuristics is the ability to recognize multiple options and reason which one is correct. ChatGPT (and all GPT models for that matter) have a single context that doesn't get compared to other possible generations. You can back-propegate and see which tokens influenced the output most, but there is no evidence that AI can conceive multiple hypothetical responses and select the most-preferred one. At least not ChatGPT.
Re:Yann LeCun: ChatGPT is 'not particularly innovative' by visarga on Thursday January 26, 2023 @03:01PM (#34537901)
Have you heard of "beam search"? The model keeps a number of optional lines of text without being greedy and picking only the most probable token every time. You can also sample with T>0 multiple times to get an ensemble of answers that can be used as input for another AI operation. You just need to do multiple interactions with the model.