Re:Why do prompts even work at all?
by PaulHoule on Tuesday March 21, 2023 @09:28AM
(#35245818)
Part of it is that you see something somewhat like the prompt near relevant text.
For instance somebody posted the other day an example where they asked GPT-4 who wrote a snippet from the Stratechery blog and it replied (after a lot of boilerplate about the difficulty of the problem) "Ben Thompson".
Somewhere in the training data it saw something like "Author: Ben Thompson" close to a lot of text that uses words in a certain way and and it learned the conditional probability distribution for that.
That underlying "knowledge" is captured by the pre-training phase where it learns the statistical regularities of text.
The ability to access that knowledge through prompts and to have a personality, add boilerplate text, be helpful, and agreeable even when it refuses to write a Hitler speech are trained in a second stage using this technique
that is, by seeing examples where somebody asks to identify the author of text it learns not only to access the knowledge it learned in pre-training the write way but to also sound equivocal about it, even though that problem is a baseball pitched right at the center of the box so far as it is concerned.
For instance somebody posted the other day an example where they asked GPT-4 who wrote a snippet from the Stratechery blog and it replied (after a lot of boilerplate about the difficulty of the problem) "Ben Thompson".
Somewhere in the training data it saw something like "Author: Ben Thompson" close to a lot of text that uses words in a certain way and and it learned the conditional probability distribution for that.
That underlying "knowledge" is captured by the pre-training phase where it learns the statistical regularities of text.
The ability to access that knowledge through prompts and to have a personality, add boilerplate text, be helpful, and agreeable even when it refuses to write a Hitler speech are trained in a second stage using this technique
https://huggingface.co/blog/rlhf
that is, by seeing examples where somebody asks to identify the author of text it learns not only to access the knowledge it learned in pre-training the write way but to also sound equivocal about it, even though that problem is a baseball pitched right at the center of the box so far as it is concerned.