The descriptions I've read about a/b testing make it sound a lot like null hypothesis significance testing, which has a well earned reputation in the social sciences. And you have to buy the things that you're testing, either from a vendor, or by employing people who assemble the stuff.