Re:Most software companies ignore user behavior
by panopticon on Wednesday August 28, 2019 @04:32PM
(#20823710)
I've also observed the problem where analysts or product managers cherry pick the data that validates their assumptions.
A "funny" example I ran into was product wanted to ship a notification that "test better" (i.e., had a higher interaction rate) in A/B testing. Some developers pointed out that the notification was likely seeing higher click-throughs because it was incredibly vague and users clicked through to figure out what it was about, and this would likely result in user frustration. Well, this insight was ignored and the notification change was shipped.
Our notification opt-out rate spiked a few weeks later.
It's an interesting question, but I think there are a few obvious easy places to get value (UX on common operations, most common error modes/corrections, etc.)
In my experience the problem is more that devs apply logging or telemetry where it is easy rather than where it is insightful. So if you didn't design the system up front to capture user behavior with some granularity you end up with a bunch of questionable proxies for it, which are hard/impossible to interpret.