This comes down to definitions. What I'm taking the article to mean is the sort of mini/micro A/B tests that many people do (taglines, button colours etc).
Using your example of "obviously good", lets say something seems "obviously good", so you launch it, and after a week, you see that it converts worse or better. That's an A/B test, where A is before the change and B is what is after. You don't need software for that, you don't need to do on-the-fly A/B testing, you can just launch.
Given a site has existed for a while, it has a pretty known A for most things you'd want to test. Doing A/B tests on small elements seems may indeed not be worth the extraneous costs of time etc.
Re:The hidden costs of A/B testing
by rangitatanz on Sunday April 10, 2016 @02:21AM
(#11465154)
There are a lot of factors that can contribute to the second week converting better or worse Eg public holidays, paydays, sporting events.. That's why ab tests run at the same time so you can compare the same population.
Most managers decide things without tests. Doing this well is easier said than done, but it's the reality of why some companies outperform others.
Exceptional growth doesn't happen because you're great at a/b testing. It happens when your team is good at decision making period.
Are they? Of course everyone thinks their ideas are "obviously good", but testing often proves otherwise. That's the whole point of it.