News for hackers. Stuff that matters.

A comment on Google ‘BigBird’ Achieves SOTA Performance on Long-Context NLP Tasks. Parent comment

Re:Google ‘BigBird’ Achieves SOTA Performance on Long-Contex... by visarga on Tuesday August 04, 2020 @03:26AM (#24047560)
The main advantage of Big Bird is its linear complexity in sequence length. If it were to be trained on the same corpus as GPT-3 what would be the advantages/disadvantages?

I am thinking maybe longer context window, faster training and less memory use, but what about performance, will it measure up?

[ Parent | on HN ]
Re:Google ‘BigBird’ Achieves SOTA Performance on Long-Contex... by phillypham on Tuesday August 04, 2020 @08:59AM (#24049466)
This is something we want to explore. BigBird just replaces the attention mechanism in BERT.

We believe something like BigBird can be complementary to GPT-3. GPT-3 is still limited to 2048 tokens. We'd like to think that we could generate longer, more coherent stories by using more context.

[ Parent | on HN ]