News for hackers. Stuff that matters.
Re:What Are Transformer Models and How Do They Work? by guy98238710 on Saturday April 15, 2023 @09:38AM (#35580532)
Why would you train the model on shorter context than you can provide? Why not provide all context you have? Sure the model has to learn to handle short context, but that occurs naturally at the beginning of the document.

Anyways, this still involves only left-side masking. Why mask future tokens when sliding window can do that?

There are no comments on this comment yet.