I think you hit the nail on the head. I prefer the look of dark mode, but because many UI elements don't support it, I stick to light mode across the board and lower brightness. There is no brightness level that is appropriate for a random mix of bright and dark mode, and trying to find one is worse than going all in on either.
But you touch on something that makes this worse; I prefer a brighter screen when in dark mode, and a darker screen when in light mode. Again, a relative thing. This is why I want consistency, above all else.