I dispute what you are saying, as someone who has studied that Lewis & Rao paper in detail. Advertising academia is pretty much just as mixed up as advertising in practice.

Here is another paper worth reading on it [0] (which cites Lewis & Rao), but provides fairly clear evidence of a distribution of outcomes of digital ad campaigns with a lot of probability mass on the side of positive lift.

The place where issues come in via Lewis & Rao is on the side of poor statistical methodology, particularly in the calculation of necessary statistical power.

But those discussions are all stuck in bad frequentist estimation techniques, and in industry there are practitioners using Bayesian methods, for example such as Gelman’s proposed “type sign” and “type magnitude” errors, with well-calibrated priors that treat the possibility of negative lift seriously.

I strongly disagree with the idea that “ads don’t work” is an “open secret” like you say. Rather, “measuring ads is extremely complicated” is the known conclusion, and marketers are just coming around to the fact that you have to invest in extremely advanced statistical methods to even know if ads are working, let alone to optimize them for an audience, especially inclusive of things like audience privacy, opt-out policies, paychological well being and so on.

[0]: https://marketing.wharton.upenn.edu/wp-content/uploads/2017/...

The open secret is that the incremental impact is small, not that it is non-existent. In experience, small enough to be questionable economically.

The more interesting work has been showing where a consumer's offline transactions are impacted by online ads.

The cited paper above, and even the influential 2018 Facebook paper by Gordon et al that tries to lobby for randomized controlled trials both disagree with your claim.

The observed incremental lift on digital conversion outcomes (particularly account signups and purchases from an online shopping cart) is (median) somewhere around 8% - 17% based on close to 500 different very large field experiments. The distribution is skewed so that the ~30% standard deviation represents mostly outcomes higher and only somewhere between 9% and 22% showed negative lift (which, as the Facebook paper points out, can also be related to the difficulty of observational statistical methods used for post facto causal inference).

8% lift is quite significant. There’s no open secret of “small impact” at all.

https://www.kellogg.northwestern.edu/faculty/gordon_b/files/...

And how much of the internet is powered by digital advertising? The web will look a lot different if that bubble bursts.

It seems like that particular article is mostly about variance and methodological concerns. I only saw the abstract though

Much.

Going back to what it was 20 years ago? I'm sold!

There is no going back, we are now talking about a 90% mobile network, that is used by some people every two minutes. This is not even comparable to the network we had. If we switch to a non digital ad world, it will be a net with different classes of citizens. I‘ll gladly pay for my 30 subscriptions but a whole global class of people will not be able to.

We'd have websites sponsored and run by people willing to pay to put them online to push their agenda.

Versus now, when we have websites sponsored by advertising and run by people trying to push their agenda.

There was basically nothing on the internet 20 years ago.

MapQuest and Amazon both started around then. People stood up personal sites at places like GeoCities. There were chat services like ICQ.

Oh boy memories.. i still remember my six digit ICQ number. It had to be circa 2000?

There were just a bunch of DragonBall webmasters trying to rank #1 for DBZ in Altavista and Yahoo, but couldnt take that spot from VegettoEX.

I vividly remember lots of things... what I wouldn't give for seeing IRC rise up again

It's still here!!!

I love IRC -- especially being able to connect with open source project maintainers. Mad respect for all of them.

There was battle.net already. I needed very little else from the internet back then.

I've been awaiting the return of marching ants and "under construction" gifs for years.

If the web went back to what it was 20 years ago that would be economically catastrophic, I mean if it happened like nearly instantly, if it happened over a space of 10-15 years OK.

In the context of the quote, let’s ignore non-ad based stuff, like amazon’s shipping network, and online banking.

In what way would resetting the ad based parts 20 years be catastrophic?

Were RSS and semi-curated news portals like yahoo and google news that much worse for publishers? (I thought they were actually better, economically, than FB...)

the context of the quote was that the web went back to what it was 20 years ago, ignoring the non-ad based stuff is cherry picking after the fact.