A/B testing is a statistical technique for teasing out the significance of changing one thing, in a world where lots of things are changing all of the time. If you just change stuff and throw it out there, you can make an educated guess about what mattered, but you don't really know and have no idea how good the data actually was.

If you want to learn more about what A/B testing is, and how to do it, http://elem.com/~btilly/effective-ab-testing/ is a good starting place.