News for hackers. Stuff that matters.

A comment on 33TB of text data for a 1T-parameter model. Parent comment

Re:33TB of text data for a 1T-parameter model by navjordj on Sunday April 16, 2023 @06:53AM (#35588901)
The original T5 models were trained with masked language modelling similar to BERT and afterwards also trained on a autoregressive task similar to GPT.
[ Parent | on HN ]

There are no comments on this comment yet.