News for hackers. Stuff that matters.

A comment on BERT: Pre-Training of Deep Bidirectional Transformers for Language Understanding.

Re:BERT: Pre-Training of Deep Bidirectional Transformers for... by zimzim on Friday October 12, 2018 @03:25AM (#18199866)
"We demonstrate the importance of bidirectional pre-training for language representations". can some one help me understand what bidirectional and pre-trained means?
[ Parent | on HN ]
Re:BERT: Pre-Training of Deep Bidirectional Transformers for... by willwill100 on Friday October 12, 2018 @04:19AM (#18200029)
* bidirectional - build representations of the current word by looking into both the future and the past

* pre-trained - train on lots of language modelling data (e.g. billions of words of wikipedia) and then train on the task you really care about but starting from the parameters learnt from the language modelling task.

[ Parent | on HN ]