Re:Understanding ChatGPT
by 2-718-281-828 on Sunday March 26, 2023 @10:35AM
(#35314864)
ChatGPT doesn't even work with "words" to begin with but with vectors encoding meaning of words. At least as far as I understand it. That's why it is able to capture meaning and concepts to a certain degree.
[ Parent
| on HN ]
It actually works with "less than words", tokens that can encode either a whole word or part of it. Example might be "you" as a single token, but "craftsmanship" might be 5-10 tokens depending on the encoder.
It has absolutely no encoding of the meaning, however it does have something called an "attention" matrix that it trains itself to make sure it is weighing certain words more than others in it's predictions. So words like "a", "the" etc will eventually count for less than words like "cat", "human", "car" etc when it is predicting new text.