RetNet was published in August (although I only learned of it last week): https://arxiv.org/abs/2307.08621
I don’t want Hackernews to be a place where people just copy and paste ChatGPT work, taking credit for it, and also potentially adding hallucinations to the ecosystem here.
When I read comments I assume they are human written, that someone put effort into that comment, and when someone is strongly asserting something (as your comment does), that they have sufficient authority to match the strength of the assertion.
ChatGPT comments always sound VERY sure of themselves, as though they are 100% right, but I know from experience that it can be very wrong, especially with more complex topics. So I think it’s poor form to post up this kind of comment with its strong assertions without saying you did a copy and paste as though it was your own original work. A modern version of plagiarism really while also lowering the overall quality of Hackernews and polluting it with generated text.
1. SSMs are a type of recurrent model that can scale linearly with sequence length, making them more efficient than Transformers. However, prior SSMs struggled with discrete sequence modeling tasks like language.
2. This paper augments SSMs with a "selection mechanism" that allows the model dynamics to depend on the input, giving it the ability to selectively remember or forget information. This makes SSMs effective on tasks requiring discrete reasoning.
3. They design an efficient parallel scan algorithm to implement selective SSMs on GPUs. Despite recurrency, this achieves up to 5x higher throughput over Transformers in benchmarks.
4. They simplify prior SSM architectures into a new model called Mamba. On language modeling, Mamba matches or exceeds Transformers of 2-3x its size, while retaining linear scaling. It also achieves state-of-the-art results on audio, genomics, and synthetic tasks requiring long-term reasoning.
This work makes SSMs truly competitive with Transformers through selectivity and efficient engineering. Mamba matches or beats Transformers on major modalities while being substantially more efficient in computation, memory, and scaling to long sequences. If replicated, it's arguably the first linear-time architecture with Transformer-quality performance!!
Can't wait to see the code!