News for hackers. Stuff that matters.
Re:Using a Markov chain to generate readable nonsense with 2... by csmpltn on Monday January 01, 2024 @08:32AM (#38831657)
> "I suppose it may help improve overall performance, you're still just blindly predicting the next word based on the previous one like a first order Markov chain would."

Instead of only taking in the last "token" as context to the function that generates the next token - take the last 15 tokens (ie. the last 2-3 sentences), and predict based on that. And that's your "attention" mechanism.

There are no comments on this comment yet.