You don’t need more color channels in all of your images to support more display primaries (at least, to a basic level). You just need GPU-type hardware built into the display which can convert each small area of an image made of pixels with colors represented in a trichromatic space to the appropriate signals sent to whatever physical arrangement of varied-primary pixels to reproduce the intended color (assuming a human viewer with typical color vision).
Probably most convenient would be to start using 16-bit floating point numbers for each of 3 color channels, so that would still increase bandwidth somewhat compared to current high-end displays which use 10 or 12 bits per channel.
It would eventually cool to have spectral representations of images, but that isn’t necessary for just expanding the gamut.
When the manufacturers get this sorted out, you'll likely still have to make a trade-off between latency (240hz), resolution (8K), HDR (bit depth), and your suggestion (more color channels)