News for hackers. Stuff that matters.
Re:Collapse of Self-Trained Language Models by corimaith on Thursday April 18, 2024 @12:45AM (#40073041)
Long Term and Short Term "memory" is a well known concept on NLP, the predecessor to Transforms were Long-Short-Term Mememory networks aka LSTMs. And LSTMs are just RNNs that attempt to solve the vanishing gradient problem. The whole strength of Transformers is that they are capable of processing the entire text corpus in parallel, hence they can "hold" long range dependencies better.

There are no comments on this comment yet.