Re:A/B testing gets misused to juice metrics in the short term
by mschuster91 on Tuesday July 12, 2022 @10:56AM
(#32070917)
The "or" was meant to be the distinction on who does the user testing - I've seen both in-house testing operations and outsourced ones. For small scale operations, it may actually be cheaper to run them in-house and only hire external testers... cameras are dirt cheap these days.
Hiring a market research company is usually worth it if you have a contract with them anyway (which gives you better rates on the testing) or lack someone on staff who knows how to deal with cameras.
Having seen lots of site redesigns go horribly wrong due to 100% earnest people trying their best and utilizing the research that was afforded to the process, I always recommending incrementally testing into changes on high-value / high-risk applications, even when the "improvements" were backed by solid research. You never know until you release.