33TB of text data for a 1T-parameter model

Re:33TB of text data for a 1T-parameter model by dougmwne on Saturday April 15, 2023 @11:54PM (#35587121)

I have yet to see anyone explain or explore why text could not be trained on multiple times. And is there any reason a text passage wouldn’t be just as good run backwards as forwards? Predicting the previous word in a sentence seems just as relevant to grasping semantic meaning as the next word.

[ Parent | on HN ]

Re:33TB of text data for a 1T-parameter model by visarga on Sunday April 16, 2023 @12:23AM (#35587251)

They might fear unwanted memorsation/regurgitation of training content.

[ Parent | on HN ]

Re:33TB of text data for a 1T-parameter model by ALittleLight on Sunday April 16, 2023 @03:23AM (#35587996)

But shouldn't they try it? Has someone tried it?

[ Parent | on HN ]

Re:33TB of text data for a 1T-parameter model by thomashop on Sunday April 16, 2023 @03:57AM (#35588108)

This is already how it works. When you train a deep learning model you show the samples of the training data hundreds of times. Of course depends on the size of your training set and your budget, etc.

You often train for 100s of epochs. One epooch means one pass over the training set.

[ Parent | on HN ]

Re:33TB of text data for a 1T-parameter model by macawfish on Sunday April 16, 2023 @12:22AM (#35587246)

One of the first transformers, BERT, was and still is bidirectional.

[ Parent | on HN ]

Re:33TB of text data for a 1T-parameter model by navjordj on Sunday April 16, 2023 @06:53AM (#35588901)

The original T5 models were trained with masked language modelling similar to BERT and afterwards also trained on a autoregressive task similar to GPT.

[ Parent | on HN ]