News for hackers. Stuff that matters.

A comment on Ask HN: Is anybody building an alternative transformer?.

Re:Ask HN: Is anybody building an alternative transformer? by nextos on Friday February 14, 2025 @03:21PM (#43052682)
The xLSTM could become a good alternative to transformers: https://arxiv.org/abs/2405.04517. On very long contexts, such as those arising in DNA models, these models perform really well.

There's a big state-space model comeback initiated by the S3-Mamba saga. RWKV, which is a hybrid between classical RNNs and transformers, is also worth mentioning.

[ Parent | on HN ]
Re:Ask HN: Is anybody building an alternative transformer? by bob1029 on Friday February 14, 2025 @03:30PM (#43052766)
I was just about to post this. There was a MLST podcast about it a few days ago:

https://www.youtube.com/watch?v=8u2pW2zZLCs

Lots of related papers referenced in the description.

[ Parent | on HN ]
Re:Ask HN: Is anybody building an alternative transformer? by RossBencina on Friday February 14, 2025 @09:46PM (#43055416)
One claim from that podcast was that the xLSTM attention mechanism is (in practical implementation) more efficient than (transformer) flash attention, and therefore promises to significantly reduces the time/cost of test-time compute.
[ Parent | on HN ]
Re:Ask HN: Is anybody building an alternative transformer? by korbip on Friday February 14, 2025 @10:14PM (#43055573)
Test it out here:

https://github.com/NX-AI/mlstm_kernels

https://huggingface.co/NX-AI/xLSTM-7b

[ Parent | on HN ]