Hi this is the author of the article. You're just making the exact mistake this article is about: you assume people who click on your ad and then buy something, bought something BECAUSE of your ad.

From the article: "Suppose Luigi’s Pizzeria hires three teenagers to hand out coupons to passersby. After a few weeks of flyering, one of the three turns out to be a marketing genius. Customers keep showing up with coupons distributed by this particular kid. The other two can’t make any sense of it: how does he do it? When they ask him, he explains: "I stand in the waiting area of the pizzeria."

It’s plain to see that junior’s no marketing whiz. Pizzerias do not attract more customers by giving coupons to people already planning to order a quattro stagioni five minutes from now."

It's the same in online marketing, you often target people who are searching for say a pair of shoes. People who are searching for shoes have way higher baseline probabilities of ending up buying a pair of Nikes, whether you show them a Nike ad or not. So if you do not correct for this 'selection bias', you have no idea what your ad did.

The research I describe in the article clearly shows (and please look it up yourself, the links are all there) that selection bias is HUGE, and that it's hard to know ROI, because true advertising effects are tiny if you measure them in an experiment.

If you're using your digital marketing tools correctly you're already correcting for this. Instead of measuring how many people click on ads, measure how many people click on ads and then compare it to how many people click on organic results to the same query. Or run ablation experiments where you stop running the ads and see what that does to your clicks & conversions. If you were making $X + $Y with the ads and then you stop running them, hold everything constant (no new campaigns, no product changes, no seasonality), and you're suddenly making $X with organic traffic remaining constant, it's very likely that you're making $Y from your ads, particularly if $Y was directly attributable to ads-related sessions. If instead your organic traffic is now making $X + $Z, it's likely that your ads cannibalized $Z in organic spending and they are actually making $Y - $Z.

Not saying all advertisers do this, but it's pretty basic data science. You should always have a control for every experiment you run.

Sure, so where the data showing that most people do this?

When I was younger, I would read over and over "don't do X" when reading about how to repair a car or write a program. And I would think, if this advice is everywhere, how could anyone miss it?

And then, over time, I learned that there is a whole culture of doing things wrong, in many areas.

How do you expect to get that data? It's buried in their Mediamath and Optimizely accounts.

Hi Jesse,

I saw a lot of youtube channels and reddit posts discussing how to optimize their ad spend (for example, https://www.youtube.com/watch?v=1O6feWOsbLs)

In the video the person starts a new online store and shows how he measures his return on advertising investment. It seems to be a pretty direct correlation between his ad spend and his increased sales.

How do you match up your thoughts on advertising with the prevalence of these kind of "advertising success stories"?

I don't think the article says advertising never works, but that people should be careful when measuring. Lets stick to the mentioned eBay example. I can totally imagine advertising for the keyword "ebay" not being worth a lot. Those people are likely to visit that site anyway. I was more surprised when "shoes" & co were mentioned. I would have expected bigger impact for those.

Hardly any of that applies to a small store with no brand awareness, since there is little/no of the "would have clicked anyway" mentioned in the article.

Another problem with measuring can be the competition. Building/Changes brand awareness (incl relative to competitors) takes time and seems unlikely to possibly be fully measured in a short experiment. Especially if the competitor currently doesn't run ads... but you not betting for them anymore might change the price and thus calculation for them.

The effects may be low but what is the alternative? You suggest there's a digital advertising bubble but to have a bubble with the assumption it will burst, you need to show that advertisers can achieve a better return by doing something else.

There's no digital marketing bubble more than there is a traditional advertising bubble. There IS however a conflict of interest in advertising that's caused by the agency which performs the advertising being tasked with showing a return on investment.

I agree that advertising as a whole might be a bubble. In fact, my next piece is about TV advertising which is arguably worse. See: https://papers.ssrn.com/sol3/papers.cfm?abstract_id=3273476

I do think it's more accepted in traditional advertising that we just don't know how well it works. In digital advertising there is a grand illusion of measurement.

Hey Jesse,

I spent 6 years working in ad-tech, and I was researching the incrementality effect of ads, selection bias and stuff like that for last 3 years. I want to let you know that your article is good. Online advertising is not a bubble, but your article perfectly explains why it is much harder than it seems.

The thing about bubbles is that nobody knows they exist. So an insider such as yourself saying, “there is no bubble,” is actually strong evidence of said bubble. Necessary but not sufficient.

This isn't really the case. Bubbles usually have plenty of people claiming they are bubbles, and people on the inside privately acknowledging it.

In online advertising there are few people on the inside who think that non-brand advertising is a bubble. It just works too well, in too many places, and too many people are making money using it. It's reliable and effective, and those just aren't attributes associated with bubbles.

Yeah, attributing everything to the last-click isn't the best measurement in the world. But as pointed out elsewhere there are techniques to avoid it.

And there are enough direct-to-consumer brands which launch on Instagram and do nothing else to be able to see how effective it is.

No, it's perfectly reasonable that everyone knows there is a bubble, but nobody knows when it will pop.

"Irrational exuberance" is a famous quote about the 90s stock market/internet bubble, and it was in 1996, when the party was just getting started.

Thanks a lot! :-)

Not sure this has much significance, but as a consumer I usually skip online ads or tune them out mentally as they are so pervasive it’s a nuisance. T.V ads work on me to an extent if I were watching live T.V. but I try again to avoid them like the plague.

Some news sites make ads agreeable if it meshes well with the content (like NY Times and such).

The bottom line I think is that there’s too much competition for my attention and it gets exhausting unless the ads are really well thought out (original ideas) or subtle—in which case the probability of me seeing the ad is less, but the effectiveness most likely increases as its less like an ad and more like relevant content

I disagree.

Most of the people I grew up with today know that it’s Miller time, that sometimes you feel like a nut, sometimes you don’t, or that coke is the real thing. They learned that from TV.

Online ads? They don’t trigger anything for me. They either need to deliver information contextually, deliver preferred placement for something I’m already looking for, or be deceptive.

> I grew up with today know that it’s Miller time, that sometimes you feel like a nut, sometimes you don’t, or that coke is the real thing.

True. I acknowledge here that anecdote is not a synonym for data, but I will offer that despite those phrases having been seared into my brain for most of my life, I don't actually drink Miller, eat Almond Joy or Mounds, or drink Coke.

If you were the kind of person who drank light beer, candy, and soda, I bet that jingle would spark something.

Ad quality these days has gone down for sure. Everything is formulaic and established and safe, yet nearly every major ad over the course of advertisement history was major because it was the opposite of these. I guess quantity has gone up to compensate.

I'm not big on beer or soda, but I do have a candy habit!

Online ad metrics tell people “what” happened, but people expect it to tell them “why.”

> but to have a bubble with the assumption it will burst, you need to show that advertisers can achieve a better return by doing something else.

To burst a bubble, you don't have to show them something that works better, you just have to convince them to stop drastically overpaying for what they've been buying.

the bubble is in the price premium of online ads

Hi Jesse,

I do growth and run experiments for a living for DTC brands. I liked this article but the research on ebay brand advertising is missing the point about competitors. If you have competitors bidding on your brand, I've seen multiple times really strong degradation on performance.

This only applies if the advertiser could be better expected to get most of the business from those searches.

If someone searches "nike shoes" they might well get them on someone else's site. Whereas someone in the pizzeria is already buying at the pizzeria.

I agree it's important to measure, but I think your concern is overblown for most companies and most non-branded searches. Assuming those companies are measuring.

This is an empirical question. I suggest you read the research I cite. For Facebook: https://www.kellogg.northwestern.edu/faculty/gordon_b/files/... For display ads: https://www.aeaweb.org/conference/2014/retrieve.php?pdfid=98... For search: https://pubsonline.informs.org/doi/10.1287/mksc.2017.1065

The problem is: yes, there's only a tiny chance somebody who searches 'nike shoes' ends up in your store. But the probability that someone ends up in your store BECAUSE he saw your ad is even smaller. So if you don't correct for selection effects your going to overstate ad effects by orders of magnitude.

Just did. One study is about brand keywords, which massively heighten the risk for what you're talking about. (I specifically excluded them in my earlier comment)

Another study is about display ads, paid for per impression. The study gives an average figure of 0.02. These are also not the typical successful google or facebook ad campaign, and are more akin to old tv ads.

The 66 page FB study isn't clear, but it also seems to be talking about ads paid for per impression? This is not what people talk about when they say you can easily track online ad returns.

Ppc ads are easy to track. You send a user to a special landing page, and you know exactly when someone got there via an ad, as only people who clicked ads go to those pages.

That said, a lot of advertisers don't do this, so they'll definitely have very real measurement problems.

That's what conversion lift studies are for. Deliberately don't show ads to a holdout group of eligible users, and measure the difference.

e.g., https://www.facebook.com/business/help/688346554927374

>It's the same in online marketing, you often target people who are searching for say a pair of shoes.

This is funny, because on HN you often see complaints about online advertising being horrifically bad because "it suggests me the products I have zero interest in", hence "online advertising sucks".

But that's the whole point! These people are complaining that they are not getting pizza coupons when they are entering pizzeria!

If it will suggest you something you are interested in already, chances are, you will buy it anyway. And if you will be suggested something strange and you will buy it, it will be incremental customer and incremental sale, and you might even become a regular client.

> If it will suggest you something you are interested in already, chances are, you will buy it anyway

For me, the key bits of advertising are a) knowing the product is available b) knowing that its specifications meet my needs and c) knowing the price. Without knowing those things, I'm unlikely to buy the product. If I know (a) and (b) and want the product but haven't bought it, then I'm probably waiting on (c).

Perhaps this is an argument for the "coupons in the pizza lobby" (or at least "coupons in the food court") type of advertising. I actually like the sorts of printed catalogs/advertising circulars that stores like Target or Best Buy provide at the store entrance, because although I may have gone there to get one thing I may end up picking up another if conditions (a,b,c) are favorable.

Magazine advertising seems to work this way; if I pick up a copy of Mac|Life magazine then I'm probably already interested in Apple products as well as compatible hardware, software and accessories. Unfortunately Mac|Life is pretty anemic at the moment (along with most of the remaining computing enthusiast print magazines) so this model seems to be in trouble for print magazines at least.

With Mac|Life the decline of third-party Apple dealers probably hasn't helped, and probably the likes of Amazon and Wal-Mart don't feel they need to advertise there. But I'd like to see more ads for software, because the Mac and iOS app stores are still pretty terrible for discovery.

Well its pretty easy to tell if its because of the advertising. Just turn it off for a while and see if it affects revenue.

That's not what the article is about - but try scaling it back to 50% and see if it results in 50% drop. The graphs show that the relationship is probably very nonlinear

Really not that easy, except if you can everything the same, aka go back in time.

Otherwise, causal inference is sadly not easy.

thats why seller should modulate buying ads with a random stream of bits, such that the impulse response (of linearized approximation in tangent space for sales / advertising dollar) can be estimated.

of course if the advertisement platform is simultaneously a dominant search engine, they can simply choose to directly modulate your discoverability through the search engine, in which case it is closer to extortion as others point out.

as a potential buyer, I'd much rather have (economical) censorship free decentralized platform where every manufacturer / seller can advertise products / parts with relevant specifications / metrics. Think DigiKey, but without a central arbiter. Obviously a lot of products that are actually superior are superior because they have a better understanding of what people need than what conventional metrics show (say not just the cheapest capacitor, but perhaps the cheapest capacitance divided by electric series resistance or whatever, for a certain niche use). This would result in placing the burden of comparing products among the manufacturers, who will also make up original but useless metric formulas with the sole purpouse of making their product rank highest. So instead of having to browse products, people are browsing metrics, and it's up to both manufacturers and end users to communicate the utility of each end metric.

Something I don't like about the parts selectors like DigiKey et al, is that I can filter and rank according to proposed properties / columns, but I can't enter a formula in terms of properties, to show me the highest ranking. I always end up having to download and scrape the full list of parts so I can perform this ranking in a spreadsheet... If their search fields allowed formulas, and if people were logged in, the manufacturers / resellers could contact you to find out what made you formulate this specific metric or loss function, which gives feedback to the manufacturers what buyers are interested in. It would also give the sellers opportunity to correct the buyer that in his or her design the buyers' metric is faulty, and point out why...

> seller should modulate buying ads with a random stream of bits, such that the impulse response (of linearized approximation in tangent space for sales / advertising dollar) can be estimated

Could you unpack this a bit? (I'm not making a point or anything, just confused and curious and not sure that googling would help me understand.)

> It's the same in online marketing, you often target people who are searching for say a pair of shoes.

FYI - you're not clearly delineating between capturing intent, prospecting, and branding. Worse, your using intent capture as a catch-all for all advertising. That's not how it works.

> Pizzerias do not attract more customers by giving coupons to people already planning to order a quattro stagioni five minutes from now."

The alternative is that the Pizzeria does zero promotion and those who are interested in "ordering a quattro stagioni five minutes from now" end up going to somewhere that does.

Ads, on Google specifically, is a zero sum game with an auction based fee structure which mostly is about capturing potential customers' intent to buy. This is the true "evil" of Google, as you basically are forced to pay a tax on all online intentions to purchase via a search from Google (it's also why Google is so scared of Amazon...hence, Google Shopping)

> and that it's hard to know ROI, because true advertising effects are tiny if you measure them in an experimen

It's impossible to know equivocally why a person makes a purchase, digital, traditional, or otherwise. Theres a lot of research on this topic but until its conclusive, we'll never have a "true" ROI.

You make it sound as if advertisers do not A/B test. My experience says they do. But YMMV I guess.

It's obviously hard to convince an advertiser that her work is meaningless.

I'm still reading the article but thought I'd mention some popup infocards (such as Audis and turtles) are cut off near the end since it's too long. At least on my desktop Chrome browser. I can read them via browser inspection but that won't suit a wider audience.

Does this apply to mobile app install ads? I just don't see how those products are accessible to consumers in the same way without the awareness driven by highly targeted, experimental ad campaigns. I'm thinking King.com or Supercell type companies.

What if advertisers jack up the cost of advertising to the point that competitors can't get a foot hold. Don't think of advertising in terms of ROI. Think of advertising as a form of extortion.

It's not the same. It's handing out coupons to people who are hungry and are looking for food; not people in the waiting area.

Google removes targeting knobs because despite what marketers think, humans consistently underperform algorithms when it comes to targeting. Typically, the people tweaking the knobs have reverse incentives because they don't want their jobs to be automated so they love these knobs. But they are just constraints on the bidding systems that end up delivering poorer performance of the campaigns.

YMMV but for the vast majority of campaigns, the algorithm does know better.

Disclaimer: I've built bidding systems for multiple ad companies, so I'm surely biased towards "the algorithm knows better".

I think it's not about whether algorithms are better than humans. It's about whose profits these algorithms optimize. Google's algorithms tend to optimize Google's profits.

Advertisers utilize algorithms too. But it gets more difficult when Google removes/limits access to information and knobs.

I agree. Third party ad companies bidding on behalf of advertisers for the best inventory have incentives more aligned with advertisers IMO (as opposed to Google and FB who tend to side with publishers since they control inventory). The main caveat is that it comes at a privacy cost (broadcasting cookie information on ad exchanges).

> Digital direct response advertising allows very accurate ROI calculations

How many ads are direct response? How many are simply clicked because they are just above the same website's entry in search results (i.e. users were already heading to that site, the ad was just there as google tax)? How many legitimate clicks on an ad that did not led to sale are discarded by google as "invalid clicks" (my adsense report has 20% invalid clicks) ? It's kind of shocking that the internet's most lucrative market does not answer those

>(i.e. users were already heading to that site, the ad was just there as google tax)?

I saw research which indicates that it is a good idea to pay the google tax. Users heading to your site may get distracted by your competitors or other stuff. The more ads are shown the less organic results (which are not easy to control) are visible without scrolling. In best case, the search results are showing your ad immadiately followed by your organic result, making up the whole screen real estate. Wherever the user clicks, he ends up on your site. Also, cost per click tends to be lower for brand search terms.

By my reading, direct marketing was LITERALLY the example used in the article that showed some of the dodgy reasoning/metrics behind common measurements of marketing success :/