Printed paper glowing white would indeed be very annoying.
Passive displays, such as old non-backlit LCD or e-ink, can be white and not annoying. Most people don't want to stare into a light source more than necessary.
Using proper brightness helps, and dark mode can help even more.
The amount of light coming to your retina stays the same..
Just lower the brightness until it matches the amount of light reflected by a sheet of paper placed next to the screen.
If you can't read the paper, turn the lights on. Being in a dark room dilates your pupils which makes them even more sensible to bright light...
I agree that it is about matching the brightness of the monitor with the background.
I will say my personal "Dev" computer uses dark modes though, I find at night with multiple monitors it's just too bright.
And it is true that, inevitably, whatever you're doing, sooner or later some white background or picture will appear on screen and blast your eyes with light. I cannot stand this, so I don't use dark themes on my PC.