So, I assume its Nvidia incompetence. Its my first and last Nvidia card in years, AMD treats users better.
Shame you don't have it around anymore, because I'd say set your desktop to like, native res, 60hz, only have one monitor installed, 8 bit SDR not 10 bit SDR, and see if the power usage goes away.
Like, on a 9800x3D /w 7900XTX with my 4th monitor unplugged (to get under the maximum super-idle load for the GPU), I'm sub-50W idle.
There are settings in the BIOS that can effect this, and sometimes certain manufacturers (Asus, mainly) screw with them because they're morons, so I wonder if you might be effected by it.
Depending on the generation, either equivalent to 4k 60hz or 4k 120hz kicks them out of idle. For my card, an RDNA3, its 4k 60hz; RDNA4 is 4k 120hz.
So, I have 4 1080p monitors... if they're all at 60hz, it properly idles. If my primary one is at 120hz, it crosses that line and is now at lowest active clockrate, so total system wattage is ~75W instead of ~50W.
I almost considered having my 9800x3D's IGPU service the trio of 60hz monitors, but its like an extra 10-15W to enable the media engine and run enough shader to run WDDM... where my 9800x3D can reach (but not cross) 80c on particularly well optimized code under a NH-D15 G2 LBC, I don't really want to throw another 15W at that.
Your monitor configuration has always controlled idle power of a GPU (for about the past 15 years), and you need to be aware of what is "too much" for your GPU.
RDNA4 and Series 50 is anything more than the equivalent of a single 4k 120hz kicks it out of super-idle, and it sits at around ~75W.