Re:MemGPT – LLMs with self-editing memory for unbounded context
by kristopolous on Monday October 16, 2023 @06:21PM
(#37907477)
That's a really recent paper. Do you actually keep up to date with everything? How do you find the time?
Re:MemGPT – LLMs with self-editing memory for unbounded context
by totoglazer on Monday October 16, 2023 @10:37PM
(#37909829)
Not to denigrate the person you’re responding to, but to add some context: That paper got a decent amount of attention already. Probably one of the more notable in the literature over the last month. Plus compared to the past year everything is slow now.
Re:MemGPT – LLMs with self-editing memory for unbounded context
by visarga on Tuesday October 17, 2023 @12:14AM
(#37910532)
Just reading a couple papers every day, the most interesting ones, and following up on reddit and twitter to get notified what people are talking about. And I am directly interested in long-context LLMs for a work related task.
I have also been dabbling with neural nets (pre-transformer), especially LSTM which have a "residual" connection, the one I was mentioning. That makes gradients better behaved. Schmidhuber tech.
The problem could be some kind of instability of attention as it scales above 10k tokens. A recent paper suggests attention mechanism needs a default value (a "sink"), and its absence produces instability.
https://arxiv.org/abs/2309.17453
Another paper says the middle part is lossy while the beginning and end are better attended.