Re:OpenAI – transformer debugger release
by bilsbie on Tuesday March 12, 2024 @07:21AM
(#39678245)
How many transformers are in a typical LLM? Or is the whole thing considered a transformer?
[ Parent
| on HN ]
The architecture as a whole is referred to as a transformer (autoregressive decoder-only architecture in the case of gpt style LLMs).
Note that there can be other types of LLMs too that are not necessarily transformer based (SSM, RNN, etc)
Re:OpenAI – transformer debugger release
by dfgtyu65r on Tuesday March 12, 2024 @07:27AM
(#39678278)
Normally, the LLM is composed of multiple transformer blocks, where each block consists of the (mutli-head) attention and fully-connected feedforward components. These are then stacked on top of each other to give the final output of the network.