> Medium sized transformer models are generally not trained with sequence parallelism, but sequence parallelism is getting more common with transformer training
Is there some word missing? You mean it's more common for large-sized Transformers?
> computation required scales faster than memory bandwidth required (square vs cube)
That is an interesting thought. I'm trying to understand what exactly you mean. You mean, computation time is in O(N^2) where N is the sequence length, while required memory bandwidth is in O(N^3)? Why is that?
In terms of difficulty of implementation it's arguably much easier than pipeline parallelism, which I'd argue is the hardest kind (at least to implement it efficiently without bubbles), and takes the most lines of code to implement (especially in Jax, where sequence parallelism is almost trivial).
For really large models, it is in fact easier to achieve peak flops because computation required scales faster than memory bandwidth required(square vs cube).