Other alternatives beyond just permuting the dimensions include strip mining/tiling and raising/sinking dimensions, techniques which come from loop nest analysis (and correspond exactly to what one would do with the loops of the code): e.g., translating NCHW -> N(C/4)(4)HW -> N(C/4)HW(4) for vectorization purposes, where we turn a 4 dimensional array into a 5 dimensional array, with the innermost dimension being a set of 4 contiguous channels which is sunk into the loop nest, and is amenable to vectorization.
Since many of these kernels are hand-tuned or generated by library vendors, there is likely not much of a choice available, but there are likely many other more optimal memory layouts out there that would require machine learning-driven compilation or mathematical optimization techniques like polyhedral compilation to explore and discover.
The most promising here being the Halide and Tiramisu compiler. Halide uses machine learning to discover scheduling and Tiramisu uses a polyhedral approach.
Also I didn't add them yet but Stanford Legion https://legion.stanford.edu/ and ETH Zurich DaCe http://spcl.inf.ethz.ch/Research/DAPP/ are more focused on memory locality and are obtaining excellent results as well.
See https://docs.nvidia.com/deeplearning/sdk/cudnn-developer-gui...
EDIT: I think https://github.com/pytorch/pytorch/issues/19092 is the RFC for this feature, and it has some more discussion.
AFAIK CuDNN always had optimizations for NCHW and that was one of Tensorflow speed issue when they choose to default on NHWC, plus the related issues on writing transformation pipelines.
So what does NHWC enables that is new?
Relevant in-depth discussion including CuDNN team lead, Julien Demouth, and Scott Gray who implemented Winograd convolution for Nervana Neon (which interestingly was CHWN so batch last): https://github.com/soumith/convnet-benchmarks/issues/93#issu...