ALBERT: A Lite BERT for Self-Supervised Learning of Language Representations

Re:ALBERT: A Lite BERT for Self-Supervised Learning of Langu... by nl on Wednesday January 08, 2020 @07:17PM (#21997231)

Summarized: by untying word piece embedding sizes from the hidden layer size + sharing parameters between layers, the number of parameters in the model are drastically reduced. They use this headroom to make the hidden layer sizes larger and given that layers share parameters, they can also use more layers without increasing model size.

This summary is wrong.

It's true that they use parameter sharing to reduce the number of parameters.

But for any given "class" of model (base/large/xlarge) ALBERT has the same size hidden layer and the same number of layers.

If you try to compare by model size (measured by number of parameters) then ALBERT xxlarge (235M parameters) has less layers than BERT large (334M parameters) - 12 vs 24 - a larger hidden layer (4096 vs 1024) and smaller embedding size (128 vs 1024).

[ Parent | on HN ]

Re:ALBERT: A Lite BERT for Self-Supervised Learning of Langu... by nl on Wednesday January 08, 2020 @08:07PM (#21997522)

they can also use more layers without increasing model size.

Additionally, in section 4.9 they compare more layers and find "The difference between 12-layer and 24-layerALBERT-xxlarge configurations in terms of downstream accuracy is negligible, with the Avg score being the same. We conclude that, when sharing all cross-layer parameters (ALBERT-style), there is no need for models deeper than a 12-layer configuration"

[ Parent | on HN ]

Re:ALBERT: A Lite BERT for Self-Supervised Learning of Langu... by danieldk on Thursday January 09, 2020 @12:59AM (#21998891)

The summary is only wrong if you want to compare the categories. I am not sure why the category names are important, except for identifying individual models.

[ Parent | on HN ]