Re:Google ‘BigBird’ Achieves SOTA Performance on Long-Contex...
by phillypham on Tuesday August 04, 2020 @08:59AM
(#24049466)
This is something we want to explore. BigBird just replaces the attention mechanism in BERT.
We believe something like BigBird can be complementary to GPT-3. GPT-3 is still limited to 2048 tokens. We'd like to think that we could generate longer, more coherent stories by using more context.
We believe something like BigBird can be complementary to GPT-3. GPT-3 is still limited to 2048 tokens. We'd like to think that we could generate longer, more coherent stories by using more context.