Re:What Are Transformer Models and How Do They Work?
by guy98238710 on Saturday April 15, 2023 @09:09AM
(#35580312)
There's one aspect I never saw explained. Why is masking used instead of a sliding window? Why even bother with masking when future tokens can be easily hidden by simply positioning the context window before the current token? Isn't sliding window optimal for maximizing context available to the model? Is masking done because moving the window would impact computational cost or output stability/quality? Can anyone shed light on this?
[ Parent
| on HN ]
Re:What Are Transformer Models and How Do They Work?
by remontoire on Saturday April 15, 2023 @09:23AM
(#35580415)
I think it's because you want to be able to predict the next token using only 1 token or the whole context window (and any size inbetween). So, you end up getting n different losses for each text snippet (where n is the size of the context window).
If i'm wrong, can someone correct here, would be useful to know.
Re:What Are Transformer Models and How Do They Work?
by guy98238710 on Saturday April 15, 2023 @09:38AM
(#35580532)
Why would you train the model on shorter context than you can provide? Why not provide all context you have? Sure the model has to learn to handle short context, but that occurs naturally at the beginning of the document.
Anyways, this still involves only left-side masking. Why mask future tokens when sliding window can do that?