Anthropic - Introducing 100K Token Context Windows, Around 75,000 Words

Re:Anthropic - Introducing 100K Token Context Windows, Aroun... by newhouseb on Thursday May 11, 2023 @04:04PM (#35907675)

Ah, you're thinking about embeddings which are basically the encoder stack on a traditional transformer architecture. Modern GPT-like models (including Claude), however, drop the encoder and use decoder-only architectures.

I could imagine something where encoders pad up to the context length because causal masking doesn't apply and the self attention has learned to look across the whole context-window.

[ Parent | on HN ]

Re:Anthropic - Introducing 100K Token Context Windows, Aroun... by revelio on Friday May 12, 2023 @02:49AM (#35912832)

Decoder only architecture? What is this? That doesn't sound like a transformer at all, are you saying gpt4 uses a totally different algorithm?

[ Parent | on HN ]

Re:Anthropic - Introducing 100K Token Context Windows, Aroun... by newhouseb on Friday May 12, 2023 @09:10AM (#35915701)

Nope, a decoder only transformer is a variant of the original architecture proposed by Google [1]. All variants of GPT that we know about (1 through 3) all roughly use this same architecture which takes only the decoder stack from the original Google paper and drops the encoder [2]

[1] Original Google Paper - https://arxiv.org/abs/1706.03762

[2] Original GPT Paper - https://s3-us-west-2.amazonaws.com/openai-assets/research-co...

[ Parent | on HN ]

Re:Anthropic - Introducing 100K Token Context Windows, Aroun... by flangola7 on Friday May 12, 2023 @12:42PM (#35918642)

How can it work without an encoder?

[ Parent | on HN ]