Unfortunately your proposed study design does not control for biases. For example, say I'm an audiophile comparing two audio formats, x and y, and I hear y is remarkably better. If I compare them side by side, my own brain might fool me that y is better. The way to control that is to do a blind test, where I won't know which one is x or y.

Also, asking people their impressions can be helpful in the human aspect of the research, but quantitatively you need some sort of metric you can evaluate their experience on. For example, you'd assign a task and see how well people did comparing the two modes, while also making sure the difference is statistically significant (meaning it wasn't just as likely to be chance).

This is just the beginning of where good study design starts. You'd also do things like assigning the modes themselves randomly, so to go back to the audiophile example, I might notice if it's always x and then y, but not if it's scrambled. You'd could go further and try to stratify the groups, so for example making sure one group isn't all elderly people and the other young. It goes on and on...

So while the scientific method is nice, especially for introducing science in educational contexts, the methodology and rationale behind research is much more deliberate and involved. By all means they can try out things themselves, but no, they will not "get better information than any study".

But that's the point. I don't care about anyone else's preferences. I'm not trying to find the best option out of the two for anyone else but me. My biases are the thing I'm trying to measure here :)

In my experience, that kind of design would be bad. I prefer light themes, but whenever I've used some app in dark mode and switch to a light one, the experience is inevitably jarring, and completely faithful to all the sun-burns-my-eyes jokes made about light themes. But it's fine the next day.

One of the more important parts of the studies referenced in the NNGroup article the blog post in the OP is referring to is that they did experiments between people rather than subjecting people to different conditions consecutively. When the initial light-dark contrast isn't present, people's fatigue ratings were about equal for both modes.

Moreover, people perceived both modes equally hard to read, but were actually more efficient in reading lightmode text.

That advantage is driven largely by the sheer amount of light - some researchers did an experiment where they cranked up the brightness on a dark theme so the experienced brightness was the same and it was just as readable as a light theme. But then you're using a crapton of electricity since you're turning dark-hued pixels to be super bright.

>It makes no sense to me to trust an unknown number of studies over my personal experience

Your personal experience is incapable of judging your performance in visual-acuity tasks and proofreading tasks, or determining the level of eyestrain caused when using light or dark mode.

Meanwhile, the scientists aren't trying to change your mind over which one is "more comfortable." They are trying to determine why you find something more comfortable and how well your the various options perform.

This is not science.

why not?