What I've seen in practice so far is that the use of telemetry has harmed software quality more than helped. It often leads developers to optimize for the wrong things and make poor design decisions. This happens because they tend to think that "the data never lies", ignoring the fact that telemetry always gives a skewed and incomplete picture.

I’ve been a product manager for products that had no telemetry, and that can be a rather undesirable place to operate, especially if you’re in the enterprise space where product changes can impact the operations of businesses.

I think it’s certainly possible to focus on the wrong things, but I don’t see that as an outcome of telemetry itself as much as an outcome of a product team that doesn’t understand the problem space or customer base.

The attributes to capture are presumably based on what teams understand to be key indicators about their app/service. I think confident incorrectness armed with bad data is just a slightly different version of a complete lack of data. Such a team was operating on whatever they imagined to be important before, and they continue to do so after, albeit with greater conviction.

But good telemetry in the hands of a good product team can be immensely beneficial for decision making and can protect customers from bad decisions. My ability to pull numbers about certain attributes has been key to my ability to shut down executive pressure to make changes that would have drastically impacted customers if not for the direct evidence that it would.

I’m also not claiming that downsides don’t exist, and privacy is always my primary concern, but there are a range of outcomes based on the maturity of a team/company, and as long as the PM understands that data is not an alternative to having a relationship with customers, I think data is pretty important.

Honestly, I can't actually remember specific examples. It's not something I dwell on. But I know that's it's happened several times that software has been made much less useful to me because features have been removed on the basis of being rarely used, ignoring the fact that even though they're rarely needed, when they are needed, they're indispensible.

More often, though, the bad telemetry-based decisions I've seen are around UI changes. Things like a laser-focus on reducing the number of clicks it takes to perform things, even though sometimes reducing the number of clicks for a thing adversely impacts the usability of it.

For bad telemetry-based UI decisions, my standout example if Firefox, although that's hardly the only one.

> But good telemetry in the hands of a good product team can be immensely beneficial for decision making and can protect customers from bad decisions.

This was actually my point of view a few years back, when telemetry started to become popular. And, as a dev who sells software commercially, I totally understand the value on that side. My experience with products that have used it, though, has shifted my view.

All that said, I do agree that it's possible to use telemetry in a way that is good for users. But I don't think it's common, and I think the reason for that is economics and human nature.

Once you start measuring a thing, that tends to become a goal rather than just a data point. And since the industry is all about maximizing velocity, that effect is even stronger. Doing proper usability studies is a slow and expensive process. Telemetry can be a useful thing as part of that process, but the tendency is to make it pretty much the entire process. That does a disservice to everybody.

> the PM understands that data is not an alternative to having a relationship with customers, I think data is pretty important.

Not just the PM. The entire company. But a relationship should be consensual, not forced. I have zero issues with opt-in telemetry. When it's not opt-in, though, it's an invasion and adversarial. I presume that's not the sort of relationship a good PM wants.

> But a relationship should be consensual, not forced. I have zero issues with opt-in telemetry. When it's not opt-in, though, it's an invasion and adversarial.

That's it. No matter how many ways you dice it - collecting data without consent or forcing opt-out is an invasion. As more and more of our lives shift to being online, our privacy and our sense of autonomy in a digital world is ever increasingly paramount.

Ideally, yes. In practice, and especially in larger shops, it’s the PM’s job to own this relationship and to make sure the important players have this understanding.

> But a relationship should be consensual, not forced.

Absolutely agree here.

A large software company in Redmond, perhaps?

Why are there code paths nobody uses, that have to be maintained, in the first place?

But how do we know the code path is no longer in use? Are people still using this CSS property (e.g., a vendor prefix)? Are people still using gopher or this one configuration variable? The more configuration options you have, the more combinations you need to test and maintain.

It can be done with error reporters like the "System program problem detected. Do you want to report the problem now?" popup in Ubuntu. In my experience, many users are willing to send error reports, and they're extremely useful, although 90% of reports are garbage.

Do you work for Boeing or something?

When I've worked on mission critical (so, safety critical, in practice), we made sure the probability of catching a failure in testing was 100x the chance of catching it in production.

Modern software development techniques like fault injection and fuzzing make this pretty easy to achieve.

We use de-identified voluntary safety reports filed by pilots, air traffic controllers, and others, along with flight telemetry data from the aircraft and other data to identify and study potential safety issues in the national airspace. Privacy-preserving techniques ensure that we can collaborate on safety and trust that the data stays non-attributional (and thus, non-punitive since participation is voluntary) despite competing interests.

We can't really do fault injection or fuzzing for real-world systems to understand, say, the impact of false low altitude alerts on risk of undesired aircraft states (e.g., controlled flight into terrain) at a certain airport.

Data doesn't always help! It can lead to assumptions, often really bad ones. And IBM isn't to be trusted with it.