Re:XLSTM: Extended Long Short-Term Memory
by zozbot234 on Wednesday May 08, 2024 @08:46AM
(#40297477)
> Specifically, the sLSTM has still recurrence (memory mixing) in it, i.e. you cannot fully parallelize the computation.
If you mean that you cannot fully parallelize inference, this might be true but also not quite relevant since the computational demands of inference are low. And you can always "parallelize" training to some extent, just by training larger batches.
Re:XLSTM: Extended Long Short-Term Memory
by korbip on Wednesday May 08, 2024 @09:56AM
(#40298173)
This was formulated a bit unclear. It is not possible to parallelize in the sequence dimension for training as it is possible for Transformers. In the batch dimension you can always do it.
If you mean that you cannot fully parallelize inference, this might be true but also not quite relevant since the computational demands of inference are low. And you can always "parallelize" training to some extent, just by training larger batches.