This is what we were doing on ChromeOS.
I'm also falling back to GL composition in some cases or while taking screenshots to avoid composing twice (HDMI + Writeback) if the scene is too complex or if other restrictions make that mandatory: Planes can only be rotated 0/180 degrees on the Pi HVS, so rotating a video to a portrait orientation is done on the GPU.
Also, you could not cache a config as valid. The configuration validity depends on the current state of the display controller. For example, if a configuration of planes on one CRTC can be set might depend on how much bandwidth is currently required by another one. I remember having to get rid of framebuffer compression on one monitor if another monitor had a resolution above a certain threshold.
Planes rotation property can be 0,90, 180 and 270, you can also flip them: https://www.kernel.org/doc/html/v4.12/gpu/drm-kms.html#c.drm.... If I remember correctly I implemented/upstreamed a few of these properties support for Rockchip display controllers.
If you can rotate a specific buffer will likely depend on your display controller plus if it's tiled or not though, since rotating a linear buffer is going to destroy BW.
If you are writing code only for one specific display controller you can look at the drivers and just figure out which configs are ok.
If your DP supports it, you don't need to GPU composite for screenshots, you can use the write back connector.
Yep. At least now the Pi's implementation does cause kernel tracebacks or lockups any more. Was rough in the beginning :-}
Flipping is supported (but not 90/270 rotation) and I use that together with a recently added transpose feature in the Writeback connector to support mirroring the primary VC4's output to the minimal DRM implementation of the official 7" display.
I'm using the Writeback connector to support screenshots, but copying every plane's configuration would be too much sometimes, so I heuristically compose some framebuffers via GL and then only place the remaining framebuffers (including the GL one) on Writeback and HDMI.
I looked at DRM/KMS briefly earlier in the year and this is what made me abandon it in the end. Can you recommend any sources of information?
The atomic API and "test only commit" both sound really useful.
There is a ton more to learn: KMS (kernel mode setting) allows fine control over the video mode in case you cannot or do not want to rely on auto-detection.
Then there's the atomic API: Unlike in the blog post, all changes applied to the output (from video mode to plane positions or assigned framebuffers...) are gathered and then applied atomically in a single commit. If necessary you can test if your atomic commit is correct and will work by doing a "Test only commit" before doing the real commit. Making changes atomically avoids all kinds of race conditions resulting in, for example, screen tearing.
Then there's the interaction with video decoding: Using FFmpeg on the Pi allows you to access to hardware decoder. They produce DRM framebuffers for each video frame. You can then directly assign them to planes and position them on the screen. The resulting playback is zero-copy and as fast as it gets on the Pi.
Another fun feature is the Writeback connector, which unlike the one ending up as an HDMI signal allows you to write your output to a new DRM framebuffer. This can, for example, be used to take screenshots of your output or even feed the buffer back into a video encoder.
One very frustration aspect is that there is basically no real documentation, especially about semantics. I guess it makes sense if you consider that there's probably only a limited number of API consumers (like desktop compositors, special video players).