The book Articulating Design Decisions (O'Reilly) deals specifically with this. It's always an uphill battle, and the book's overall lesson is "empathize as much as possible", but it's a great read. It has definitely helped me in dealing with more difficult management types who don't understand the design or technical side of product building.

Thanks, I have added it to my reading list.

Hey there, I wrote Articulating Design Decisions. Convincing stakeholders of your decisions is critical to the success of every project for sure... I've been there many times.

You can watch my talk from MidwestUX last year that gives some practical tips too: https://vimeo.com/110260257

The early release eBook is available now. The print version will be available in August.

Great read. Tom is a master when it comes to clear communication.

would A/B testing, in some capacity, the decision in some form or another, help with this?

It could make it worse. Just look at Amazon's homepage.

I don't think so. It would certainly help in some specific cases and some ideas are certainly worth testing out. However : -A/B testing can only answer some very specific questions. Good UX is not just measured by engagement. -some of these decisions are just bad practices that are widely recognized : for example we were asked to add modal popups pretty much everywhere. -we are severely shorthanded and implementing these changes and the A/B tests can be a very significant time investment. Especially when they are abandoned before going live for the reasons given by everyone but the person in charge.

Think of the eternal vastness of configuration spaces (even in domains as restricted as, say, static HTML/CSS pages) and then you know why AB testing is not a solution to everything. (Multivariate testing, too. It’s just slightly more elegant in some respects, but suffers from the same basic deficits and additionally has its own issues.)

Time is limited and traffic is limited. You simply cannot test everything, in fact, the amount of variants you can actually test during, say, a year may be shockingly low compared to the amount of variants possible (at least if you are not Google – but most aren’t).

Without a strong sense of direction and discerning eyes and hands that provide plausible variants to test, AB testing can be quite useless. AB testing is a great tool to verify hypotheses (and those may well be competing hypotheses), but I really question its usefulness as a creative tool.

If you can plausibly only do a couple of AB tests a year (the possibility of running an AB test being a function of traffic and time) you better not waste that opportunity on button colors or some similar nonsense.

Sure, evolution through natural selection produced some awesome designs and is basically a completely blind process based purely on testing of different variants with only slight and random differences with no involvement of any designer at all (sound familiar?) – but it also took billions of years. Most designers just don’t have that kind of time …

To more directly answer your question, AB tests can certainly be used to defend against bad hypotheses. If there is a culture of testing and new designs have to be battle tested then that’s less of an opportunity for someone in a position of power to screw things up by fiat. However, that’s also a tremendous waste of time and efficiency if bad designs that could be sieved out quickly by simple heuristics have to also first be tested, which is just a slow process and a wasted opportunity to test something better.

Maybe think of AB testing as a last line of defense, but be careful, AB tests can be paralyzing. (Or rather, first AB tests paralyze everything, then the boss is annoyed by how slow everything is going and changes everything by fiat and immediately without any testing at all. From one extreme to the other …)