Re:Dear Websites, Stop Asking for Ransom Sign-Ups
by naijaboiler on Thursday August 03, 2023 @07:06PM
(#36993312)
If this measurement was done by product or marketing people, my lived experience was that the experiment design was flawed, the conclusions they reach are likely not correct ones.
But I have seen other decisions made based on proxies to "paid us money." Eg we find that X% people who perform action Y wind up paying us money, and for really long funnels it's easier to get statistical significance for action Y than for "paid us money." And so we make some change, find that it increases the rate at which people do action Y, and declare victory. We try to keep in mind potential downstream impacts (like loss of trust), but it's hard to argue a vague, potential downside against a solid, measured number sitting in a google doc in front of you. I suspect the phenomenon you call out is somewhat common.