IME this is the way to go; take an ensemble of big, accurate models, then distill them down to the smallest model you can get away with. There are really good tricks in this paper
This summary is wrong.
It's true that they use parameter sharing to reduce the number of parameters.
But for any given "class" of model (base/large/xlarge) ALBERT has the same size hidden layer and the same number of layers.
If you try to compare by model size (measured by number of parameters) then ALBERT xxlarge (235M parameters) has less layers than BERT large (334M parameters) - 12 vs 24 - a larger hidden layer (4096 vs 1024) and smaller embedding size (128 vs 1024).
Additionally, in section 4.9 they compare more layers and find "The difference between 12-layer and 24-layerALBERT-xxlarge configurations in terms of downstream accuracy is negligible, with the Avg score being the same. We conclude that, when sharing all cross-layer parameters (ALBERT-style), there is no need for models deeper than a 12-layer configuration"
It's worth noting that TPUs are available for free on Google Colab.
Yes, and you can also get a research grant, which gives you several TPUs for a month. But that does not mean that you can easily deploy TPUs in your own infrastructure, unless you use Google Cloud and suck up the costs (which may not be possible in academia).
It's common knowledge not to bother running a deep model like ALBERT, BERT, or XLNet, etc without at least a GPU. Training and inference with models of this size on CPU's is typically considered to be intractable.
Obviously, this paper is from Google, so they have free access to TPU's, which is arguably optimal for an arbitrary TensorFlow model in terms of performance, but if you don't have access to TPU's, the clear choice is to use GPU's instead. It won't be quite as fast, but it is sufficient and GPU's are relatively cheap and available.
It's pretty common to run inference on CPUs. There are lots of operational and cost reasons why this makes sense in at least some cases.
I am not sure if this is common knowledge, and if it is, it is wrong. With read-ahead and sorting batches by length, we can easily reach 100 sentences per second on modern CPUs (8 threads) with BERT base. We use BERT in a multi-task setup, typically annotating 5 layers at the same time. This is many times faster than old HPSG parsers (which typically had exponential time complexity) and as fast as other neural methods used in a pipelined setup.
In section 4.8 they compare accuracy at the same amount of training time for the biggest of each model and show that ALBERT is substantially better.
Huggingface did just that: https://medium.com/huggingface/distilbert-8cf3380435b5
Another interesting finding is that this speeds up training due to the smaller number of parameters.
However, my worry is that those of use that do not readily have access to TPUs will get even slower models when using CPUs for prediction due to the additional and wider hidden layers. (Of course, one could use ALBERT base, which still have 12 layers and a hidden layer size of 768, at a small loss.) Did anyone measure the CPU prediction performance of ALBERT models compared to BERT?
Edit: I guess one solution would be to use a pretrained ALBERT and finetune to get the initial model and then use model distillation to get a smaller, faster model.