I have worked for some pretty arrogant, business types who fancy themselves “data driven“ but actually they knew nothing about statistics. What that actually meant was they forced us to run AB tests for every change, and when the tests nearly always showed no particular statistical significance, they would accept the insignificant results if it supported their agenda, or if the insignificant results were against their desired outcome, they would run the test longer until it happend to flop the other way. The whole thing was such a joke. You definitely need some very smart math people to do this in a way that isn't pointless.