Building a Better Search Engine for Semantic Scholar (2020)

Re:Building a Better Search Engine for Semantic Scholar (2020) by vinni2 on Saturday August 28, 2021 @08:01AM (#28337392)

It depends on what the BERT was pre-trained on. If it was pre-trained on Google News Collection like most Transformer models are, fine-tuning on something like scholarly data won’t perform so well due to vocabulary mismatch. Besides as the author mentions training models like LightGBM are efficient and perform reasonably.

[ Parent | on HN ]

Re:Building a Better Search Engine for Semantic Scholar (2020) by spratzt on Saturday August 28, 2021 @08:27AM (#28337508)

I agree with this. In my experience, LightGBM is often good enough. The additional improvement provided by neural net models is not worth the substantial increase in development time.

[ Parent | on HN ]

Re:Building a Better Search Engine for Semantic Scholar (2020) by dunefox on Sunday August 29, 2021 @01:25PM (#28348638)

> pre-trained on Google News Collection like most Transformer models are

BERT for example was trained on Wikipedia and Book corpora. Do you have a source for this claim?

[ Parent | on HN ]