Re:Apple's products are getting harder to use, ignore princi...
by catalogia on Sunday December 22, 2019 @06:55PM
(#21860499)
Be careful with data-driven design, you can easily end up fooling yourself into seeing evidence in the metrics for whatever you want to be true, I've seen it happen plenty. Accidental p-hacking by well meaning but misguided "data driven" programmers is not at all uncommon.
Furthermore, in my experience the data doesn't necessarily mean what it's believed to mean, with the favored explanation for the data being assumed to be the correct explanation. (e.g. "We want more engagement. This data shows users are staying on this page longer, that's a measure of engagement." It may instead be a measure of your users getting confused, lost, etc.) My point here is that an earnest attempt to be data driven doesn't eliminate the matter of bias. Being cognizant of possible bias when evaluating the data is important, otherwise you'll have a false sense of confidence.
Furthermore, in my experience the data doesn't necessarily mean what it's believed to mean, with the favored explanation for the data being assumed to be the correct explanation. (e.g. "We want more engagement. This data shows users are staying on this page longer, that's a measure of engagement." It may instead be a measure of your users getting confused, lost, etc.) My point here is that an earnest attempt to be data driven doesn't eliminate the matter of bias. Being cognizant of possible bias when evaluating the data is important, otherwise you'll have a false sense of confidence.