ALBERT: A Lite BERT for Self-Supervised Learning of Language Representations

Re:ALBERT: A Lite BERT for Self-Supervised Learning of Langu... by zuzun on Wednesday January 08, 2020 @12:28PM (#21993606)

Is there a speed-up? In their paper in table 3, once you compare each ALBERT model with the smaller BERT model, you're looking at similar accuracies and longer training times.

[ Parent | on HN ]

Re:ALBERT: A Lite BERT for Self-Supervised Learning of Langu... by nl on Wednesday January 08, 2020 @08:02PM (#21997495)

They are comparing the speed to execute training to 125K steps, not speed to a given accuracy.

In section 4.8 they compare accuracy at the same amount of training time for the biggest of each model and show that ALBERT is substantially better.

[ Parent | on HN ]