Google ‘BigBird’ Achieves SOTA Performance on Long-Context NLP Tasks

Re:Google ‘BigBird’ Achieves SOTA Performance on Long-Contex... by phillypham on Tuesday August 04, 2020 @09:05AM (#24049531)

I think the insight is not incredibly original and follows naturally from OpenAI's Sparse Transformer. The idea is similar to Longformer. Two teams at Google had a similar insight, hence the high number of authors.

The original implementation only took a couple of months and was primarily motivated by internal Google applications. Natural Questions was the first external benchmark we tried to validate on, which took a few months to find the right setup. All the other datasets, took a few weeks but the effort was done in parallel given the large team.

There was quite a bit of frustration dealing with Tensorflow, TPUs, and the XLA compiler that maybe set us back a few months, too.

[ Parent | on HN ]

Re:Google ‘BigBird’ Achieves SOTA Performance on Long-Contex... by phillypham on Wednesday August 12, 2020 @10:07PM (#24139269)

Edit: This is not meant to criticize the Tensorflow, TPU, or XLA team. They responded quickly to our bugs and made the work possible. I just meant there was some extra organizational overhead.

[ Parent | on HN ]