It depends on what the BERT was pre-trained on. If it was pre-trained on Google News Collection like most Transformer models are, fine-tuning on something like scholarly data won’t perform so well due to vocabulary mismatch. Besides as the author mentions training models like LightGBM are efficient and perform reasonably.

I agree with this. In my experience, LightGBM is often good enough. The additional improvement provided by neural net models is not worth the substantial increase in development time.

> pre-trained on Google News Collection like most Transformer models are

BERT for example was trained on Wikipedia and Book corpora. Do you have a source for this claim?

Funny. I was searching hotel reviews today and Google doesn't even know that "wifi" and "wi-fi" are the same thing. (I'm sure the Google reviews use a different algorithm than the main Google site; but it was still funny to see with all the NLP stuff they are doing)

Like many organizations, there is often a disconnect between google research and production.

It’s likely the models run in google search are Bert inspired, or were compared with Bert derived models.

Restrict to a small number of passages.

Run on TPUs.