Re:Code Walkthrough of Bert with PyTorch for a Multilabel Cl...
by yashvijay on Wednesday August 14, 2019 @05:02PM
(#20699922)
Since the training is done on the tasks of masked word prediction and contiguous sentence prediction, I'd suggest about a million sentences (from the same domain), with an average token length of 7 per sentence. Longer sentences would definitely help, as BERT uses the transformer encoder architecture which has multi head attention. This would enable the model to do better contextual representation learning for the embedding layer.
[ Parent
| on HN ]
There are no comments on this comment yet.