And by the way, sure one can probably figure out whether it's a button or not after some time of careful analysis. The problem is that if instead of it taking me 10ms to figure out it's a button I have to pause and understand it and it takes me 10 seconds, I get annoyed.

I have this problem right now with Firefox Proton. Since it has problems respecting my color scheme, it takes me some time to distinguish (in an inactive-colored titlebar) which is the damn currently active tab. Before, I would be able to do it almost subconsciously, without distracting me from my current goal. Now, my train of thought has to stop and search for the damn button, then I have to restart my thoughts. And we all know how that hurts.