Re:Using ChatGPT as a Co-Founder
by yunyu on Monday December 05, 2022 @08:18AM
(#33865208)
Most LLMs are crippled at tasks regarding character level manipulation because of how the inputs are tokenized (words and numbers are split into common substrings rather than the original characters, 101 -> “10” + “1” vs “1” + “0” + “1”).
If you tried this binary conversion task with a model that takes in raw bytes (for instance, ByT5) it would perform almost perfectly. In GPT’s case, there’s a trade off between having a long context window vs being good at character level tasks, and OpenAI picked the former.
Re:Using ChatGPT as a Co-Founder
by gdw2 on Monday December 05, 2022 @01:53PM
(#33870036)
I tried to use it for help with a wordle. I think the word was EJECT and I had E?E?T. It seemed to understand what I was asking, but came up with 5 and 6 letter suggestions that didn't fit.
If you tried this binary conversion task with a model that takes in raw bytes (for instance, ByT5) it would perform almost perfectly. In GPT’s case, there’s a trade off between having a long context window vs being good at character level tasks, and OpenAI picked the former.