Let the "framebuffer chip" handle the timing sensitive HDMI/DVI dance, and then you can use the general-purpose controller for things like "map a more well-supported graphics protocol to this language". If the controller fails to meet a timing requirement, the framebuffer chip doesn't cause the monitor to lose synch, you'd just get a torn image because it wasn't fully populated in time for the frame to render.
What I've been waiting for is some sort of "modern MCU as replacement for expensive, clunky, slow old video cards." It seems like if you want to use something like a Pi Pico, it would use almost all of the resources just to maintain synch and timing, so it might not be amenable for the other half of the job-- monitoring the external bus and waiting for inquiries on the framebuffer/firmware/support register address ranges.
The Pi Pico has 2 cores so while one core is used entirely to maintain the display the other core can do other things.
Analogue output is actually easier that's why it's done. Here, you just have to push out some signals at the right (slow) timing and you get a picture. DVI/HDMI requires pretty decent technology to do effectively.