https://gist.github.com/sighingnow/ce6aef69aaa379517ac4b68bc2873a9c

Visualizing how pipeline mode parallel works and how batches are scheduled under various scheduling scheme: GPipe (F-then-B), Pipedream (1F1b), Pipedrea-2BW (no flushes between iterations) and Eager-1F1B (better computation-communication overlapping).