News for hackers. Stuff that matters.

A comment on DeepNet: Scaling Transformers to 1k Layers. Parent comment

Re:DeepNet: Scaling Transformers to 1k Layers by londons_explore on Wednesday March 02, 2022 @06:34PM (#30535012)
It is notable that they got only 2.5 extra BLEU while translating text to english, and 6.2 extra when translating text from english to another language.

Since the network will have seen far more english text than text of other languages, it suggests that performance on limited training data is more improved.

[ Parent | on HN ]
Re:DeepNet: Scaling Transformers to 1k Layers by modeless on Wednesday March 02, 2022 @07:19PM (#30535405)
That's exciting. Getting better performance with more data is trivial, the hard part is getting better performance with less data.
[ Parent | on HN ]