News for hackers. Stuff that matters.

A comment on XLSTM: Extended Long Short-Term Memory. Parent comment

Re:XLSTM: Extended Long Short-Term Memory by YetAnotherNick on Wednesday May 08, 2024 @03:54AM (#40295452)
No, it means dividing the sequence into multiple chunks and processing them one by one, very similar to recurrence. See [1]. Sequence parallelism is needed when the sequence can't fit in a single GPU. Sequence parallelism is the hardest parallelism, but it is required for longer sequence. Many models just trains for smaller sequence length for majority of the training and switch to sequence parallelism for last few percentage of training.

[1]: https://arxiv.org/pdf/2105.13120

[ Parent | on HN ]
Re:XLSTM: Extended Long Short-Term Memory by logicchains on Wednesday May 08, 2024 @04:15AM (#40295568)
>Sequence parallelism is the hardest parallelism, but it is required for longer sequence

In terms of difficulty of implementation it's arguably much easier than pipeline parallelism, which I'd argue is the hardest kind (at least to implement it efficiently without bubbles), and takes the most lines of code to implement (especially in Jax, where sequence parallelism is almost trivial).

[ Parent | on HN ]