Anthropic - Introducing 100K Token Context Windows, Around 75,000 Words

Re:Anthropic - Introducing 100K Token Context Windows, Aroun... by clbrmbr on Thursday May 11, 2023 @09:00PM (#35910522)

Makes me wonder whether we could get really huge contexts much more efficiently by feeding back a higher layer back into the tail end of the model. That way it has a very clear picture of the recent text but only a compressed picture of the earlier parts of the document.

(I think I’ve got to read up on how transformers actually work.)

[ Parent | on HN ]

Re:Anthropic - Introducing 100K Token Context Windows, Aroun... by desperate on Thursday May 11, 2023 @10:09PM (#35911049)

Afaik you're describing something akin to a recurrent neural network, and the problem with that is that it doesn't parallelize well to modern hardware. And vanishing gradients.

[ Parent | on HN ]

Re:Anthropic - Introducing 100K Token Context Windows, Aroun... by jiggawatts on Thursday May 11, 2023 @11:06PM (#35911405)

I had the same thought as the comment you're responding to.

Recurrent neural networks are bad when the recurrence is 100x long or more. You need long chains because with a token-at-a-time, that's what you need to process even one paragraph.

But if you use an RNN around a Transformed-based LLM, then you're adding +4K or +8K tokens per recurrence, not +1.

E.g.: GPT 4 32K would need just 4x RNN steps to reach 128K tokens!

[ Parent | on HN ]