Re:People tricking ChatGPT “like watching an Asimov novel co...
by visarga on Friday December 02, 2022 @02:32PM
(#33835616)
> I really want to know how it works.
You may be disappointed to know that the exact inner workings of the model are still largely unknown. We understand the basics of how it works, such as how changes in the model size and data size will affect its performance, or how to combine various supervised datasets to train it to solve tasks and what the model probabilities are supposed to mean, but the complexity of the model is such that it cannot be fully simulated or imagined. It is similar to the workings of a cell, the brain, or even a protein - we know the basics, but the full complexity of it is beyond our current understanding.
The true complexity is in the weights, and in the dynamic activation patterns. This excellent visualisation article will show you the state of the art in interpreting transformers.
You may be disappointed to know that the exact inner workings of the model are still largely unknown. We understand the basics of how it works, such as how changes in the model size and data size will affect its performance, or how to combine various supervised datasets to train it to solve tasks and what the model probabilities are supposed to mean, but the complexity of the model is such that it cannot be fully simulated or imagined. It is similar to the workings of a cell, the brain, or even a protein - we know the basics, but the full complexity of it is beyond our current understanding.
The true complexity is in the weights, and in the dynamic activation patterns. This excellent visualisation article will show you the state of the art in interpreting transformers.
https://www.lesswrong.com/posts/mkbGjzxD8d8XqKHzA/the-singul...
https://www.youtube.com/watch?v=pC4zRb_5noQ