Re:Using GPT-3 to pathfind in random graphs
by joe-collins on Saturday September 24, 2022 @03:32AM
(#32960453)
I think the sentences in this example are, odd vocabulary notwithstanding, fairly typical descriptions of logical implication, which I expect has non-trivial representation in the training set. GPT-3 is probably capable of matching the pattern of the tokens (sentence structure), even if the content of the tokens (spelling/vocabulary) is novel.
[ Parent
| on HN ]
Re:Using GPT-3 to pathfind in random graphs
by uup on Saturday September 24, 2022 @08:21AM
(#32961833)
I doubt it. There are billions of parameters and weights all combining in non trivial ways to produce the output. I don’t think you can describe any of its workings with simple terms like “doing token substitution” even if that’s what it looks like to us when we look at the output.
Re:Using GPT-3 to pathfind in random graphs
by joe-collins on Saturday September 24, 2022 @01:23PM
(#32964711)
Try it. Reduce the tokens to single characters but keep the structure. Does it work? Maybe single characters looks too much like an abbreviation and that throws it off, because those are found in sufficiently different contexts. Exchange to two characters, three. Throw similar inputs at it and find the point where it reliably "gets it".