This is because businesspeople like to fiddle with formulas, graphs, and filters untill they isolate (i.e. cherry-pick) the subset of data that [makes them look good | supports their agendas]. They don't want canned reports. Excel makes this easy because they already know how to do those things in Excel.

People get really mad when you call this p-hacking, btw. I used to try and make this point to ”analysts” that I worked with, that science involves coming up with a hypothesis and then looking at the data. The response I got often enough that I’m sure it’s being taught somewhere was that you “need to let the data tell it’s own story”.

Turns out lots of people have decided what the conclusion is and are taking that conclusion and the data, and then blindly munging until they get a path which connects them.

As my professor said: All good statistics is done before you have looked at the data.

But where do you get your hypothesis from?

your hypothetical model of reality.

But that's a lot of work so businesses don't want to pay for it.

Whatever orifice you desire.

True, for clean data. You can’t clean data without looking at it (a lot), though.

If you have data that's so 'dirty' that you can't decide on the filtering rules in advance (or based on only historic data), then what you have is garbage, not data. Therefore, we could call the art of shaping this into meaningful stories garbage science.

Tell me in a comment you have never worked with business data in your life.

Business data is full of minor inconsistencies which are not obvious until you sit in front of it. Products are sold by different units. Reporting ranges and aggregates are slightly different. Subsidiaries use categories which are close but not exactly identical.

There is generally plenty of massaging to do before you can get the information you need.

> I’m sure it’s being taught somewhere was that you “need to let the data tell it’s own story”.

AWS is pushing this message on every NFL broadcast with their Next Gen Stats ads. "The data tells us all"

I think a lot of times unless you are operating at hyperscale and bps matter, most business data analysis is more art than science. Ex, if you have < 200 sales in a month and you want to report this, it's totally normal to want to pull it into excel and fiddle with the numbers, make ad-hoc corrections, divide those 200 sales up in very subjective ways, etc. The smaller the company the more the reporting layer is only a weak approximation of reality, there isn't enough there to create a narrative around. "Hypothesis testing" such data is honestly a waste of time.

Of course you want to be intellectually honest about it, and I agree that cherry-picking the "right" data can be a big problem in the wrong kind of organization. I remember one time a Jr analyst I managed was asked by the CEO to create a certain chart. I was not looped in. The CEO then used that chart to convince himself and many others on the executive team to make a huge product change that ended up being a complete disaster.

Excel also makes it easier to experiment. Pandas requires you to change an entire column which can frequently be processor-intensive, take forever, and be hard to understand. With Excel, you can just take a random cell, try something out, and if it comes up with nonsense, undo it

Why not df.head()… ?

That can also take a while, and it isn’t as easy to view the values side by side. Excel is simpler (although way less capable) a lot of the time

It is sometimes more possible to determine if your analysis is wrong in excel because you can look at the numbers and the numbers have meanings. It can be harder to debug things if you can’t see intermediate computations. I guess I would say that excel prevents some kinds of bad problems with faulty analyses but opens you up to other problems like overfitting or the cherry-picking you describe. But it also seems impossible to produce a reporting tool that both gives useful analyses and doesn’t do incorrect analyses.

As one of those business people, I find I’m not very good at requesting reports. There is a gap between what I think is interesting and what actually is interesting. I use spreadsheets (and sometimes visualization tools) to help bridge the gap and make a better on going request to our reporting team.

Often, this is learning for me. I have a bunch of Stephen Few’s books and use exported CSV files to figure out which reports are useful to me and my org. When I find them, I do make a request for standardized reporting. These often become the basis for regular review meetings. In those meetings, we still come up with instances where we need to export to CSV and get into the data to understand what we are looking at.

Our work is going through a big shift this year that means our historic data is not helpful in predicting trends. That’s increased the need for this kind of engagement with the underlying data.