I've said it before and I'll say it again - if your business exists primarily to help users get a task completed and move on, then you absofrigginglutely should not be running A/B tests the way content sites run them. If you can achieve greater click per user or time on site metrics by making your product worse at its core function, then you need to have analysts that know what they're doing in those situations, and they need to be able to win in fights with the business types that don't understand this stuff and assume higher traffic or CTRs are automatically a win. You're not playing the same split testing game as everyone else, and you have to have people with both the expertise and (equally important) the authority to handle this special situation properly.