News for hackers. Stuff that matters.
Re:What We Know About LLMs (A Primer) by janalsncm on Tuesday July 25, 2023 @08:38PM (#36871668)
> Transformers can read the entire sequence at once and learn to “pay attention to” only the values that came earlier in time (via “masking”)

Unless the text fits into the model’s context window, this is incorrect. The self-attention layers will train via a sliding window over the text.

Learning to attend only to previous tokens is also not correct. There are a lot of ways to train a transformer. BERT is bidirectional for example.

Re:What We Know About LLMs (A Primer) by wilhelm____ on Wednesday July 26, 2023 @08:36AM (#36876622)
I mention that above this sentence in the image - glad someone was "paying attention to" the text ;)