> Show ads to your test group, don't show to control

Which is the wrong experiment to run. The old adage is that 50% of advertising works, you just don't know which part. You need an experiment to prove that ad spending scales with ROI, as ad agencies claim. That may well be completely false.

That’s the thing - you run these test with individual campaigns in many channels. Digital gives you the ability to test this at scale as you grow your advertising, thus you can gauge with high confidence exactly what advertising is scaling with positive ROI.

Incrementality testing isn’t limited to your advertising as a whole - you can get down to very fine detail as long as your experiment is correctly designed and you wait for stat-sig results.

The basic experiment can be varied to allow for that.

Specify a set of zones. Target different levels of advertising at these. Look at sales trends. Rinse, wash, repeat.

Note that major corporations often designate specific areas for product testing. The Chicago area, and Australia, see trial runs of McDonalds products and store concepts, as an example.

The fundamental concept of testing advertising effectiveness is not new. Here's an account from 1909 (from one of my faviourite sources on advertising and media, Hamilton Holt's Commercialism and Journalism):

https://archive.org/details/commercialismjou00holtuoft/page/...

> You need an experiment to prove that ad spending scales with ROI, as ad agencies claim

There is a limited audience that can be influenced. Just like house ads on a site (where you advertise to sell your own goods on your own site), there's only so much that is affected because you have finite traffic. There is a curve for every permutation of context (demographic, geographic, etc).

> Take an audience of X people. Divide them in two. Show ads to your test group, don't show to control. Watch your business grow and gauge the lift between the two audiences.

So advertising industries use people as guinea pigs to study the "real effect of their ad dollars". No ethics committee, no informed consent, nothing.

Yet another reason to block all ads.

If you can ethically do something (show ads) or not do something (not show ads) it's equally ethical to run an experiment where you do the thing some of the time and measure the effect.

Internet companies run A/B tests all the time, and it's not controversial.

Hyperbolic.

Isn't that literally the test described by FB economists in the article that showed that selection effect was 10x the impact of advertising?

Agree. I have used the term "incrementality" to communicate the incremental benefit of a given ad compared to a control whereas the article uses "advertising effect". Measurement is also difficult because every ad platform grades their own homework (they each have analytics that attributes sales to their platforms liberally).

It is important for a brand to have their own attribution platform that measures incrementality as well as the relative contribution each ad exposure makes toward a given sale.

Both of these things are possible to measure. The problem is that the band of users that display positive ROI (when incrementality is measured) is so small that people don't believe the data.

Additionally, brand exposure is hard to measure and almost completely ignored in digital advertising. Over time I imagine that there will be more research in this area (beyond just measuring an in-view ad impression)

How do you separate "people who haven't seen an ad" into a group in Google analytics? I don't believe "organic" source does that, because that means they weren't targeted by the ad in the first place, creating a bias.

Advertisers have many tools to measure "ghost ads" aka "people who haven't seen an ad but would otherwise have qualified for all your target criteria". See https://papers.ssrn.com/sol3/papers.cfm?abstract_id=2620078 as an example.

That's of course possible to measure in theory. I meant it as a response to "Facebook, Google and others make these tests possible in their platforms" since I haven't seen it exposed as an explicit option on either.

Then you haven't been talking to the right ad sales teams. Google and Facebook have entire product options built around this specific concept. You can run a "ghost ads" test on Google for as little as $20k in ad spend; it's relatively accessible even for smaller players.

For Facebook, Reach ads have a brand lift objective that will estimate for you. If you spend large amounts they have a measurement team that will run the actual study for you. Source: I'm running one right now for a CPG brand.

You can run this yourself without any help from them: show one group ads for your site, and show another group white rectangles. Compare performance.

Facebook has this exactly this tool, aka Conversion / Brand Lift [1]. It's a real experimentation system, not a "simulated" one like Google's Ghost Ad.

[1] https://www.facebook.com/business/help/688346554927374

Yep. Lift tests are solid gold. A lot of people pan FB ads but the reality is they can be very incremental.

The flip side is that not every campaign, targeting strategy or ad will be incremental. You have to test and retest your way to it. A blanket assumption won’t work.

Off the top of my head, you could do geographic bucketing. Take say 100 cities, and only run ads in 50 of them.

Geo tests are useful for sure, but also require a lot of data to get statistically significant results. Any noise in your data (seasonality, PR, etc) has to be taken into account as well. They’re harder to execute in general but can work well.

The interest is more on markets than individuals, in general, so what you're measuring is market response given advertising exposure, or more critically, ROI given spend.

The fact that Web and AdTech allow the appearance of individual-level targeting does not necessarily make that a Good Thing.

I'm running a campaign that spends about $10,000 a day for a brand on Facebook. In fact at that level they will run said brand lift study for you. So yes, despite this articles implication the data does exist and we do know the difference between advertising to an audience and not.

That's millions per year. What general class of industry are you in?

The average "target" client on the lead list of your average big name advertising agency in a European country with a population of 50 million will have a yearly spend of 50-100 million across all channels, with 1-10 million going on agency fees excluding 3rd party costs. 10k per day on Facebook is a small consumer start-up levels of adspend.

1. In the grand scheme of advertising it's actually still a fairly small budget. 2. For this advertiser they really only advertise q4 so it isn't quite that much. 3. It's a CPG in a grocery store is all I can really say.

Acquiring an audiences to divide into a split test is the hard part.

It is something that could work for retargeting existing users but doesn't work for acquiring new visitors.

What makes you believe this to be true?

You define an audience. You split it. You test and measure the result.

What component of that strategy doesn't work for new visitors? Many eyeball vendors (Google, Facebook, heck traditional TV and other media!) are set up precisely to allow for such cases.

Not quite.

One tool with FB lift tests is to make a lookalike audience. FB will then split that audience in two for you so you have a test and control group.

The original test required two ad accounts and keeping your overall advertising static (not changing anything) for the duration. Today it is much easier and has a lot more features.

Google does a search lift test with YouTube as well. You can run a large set of ads against an audience in YT and google will either suppress them for a subset. Then you can see your overall organic lift based on your video ads.

FB can track ads using clicks and view through. With a control group, you can only use view through. I am just skeptical of how it is a fair test especially when FB has an interest to prove that ads work.

YouTube brand lift is an indirect measure that is hard to translate to actual sales to see if the ads were cost effective.

The math is fuzzy when measuring marketing ads and it is real hard to be truly scientific.

The proper test is to use viewthrough for both the experiment and control

The concept, principles, and methods, are taught in the context of marketing and advertising academic programmes.

See, e.g., ADVT5440 and ADVT5501 at Webster University, for example:

This course emphasizes the importance of critical thinking in the planning and development of message strategy for advertising and other marketing communications tools. Class discussions explore the decision making process and development of criteria for evaluation of alternative message strategies.

http://www.webster.edu/catalog/current/graduate-catalog/cour...

Or in related textbooks, e.g.,

https://www.worldcat.org/title/marketing/oclc/61458339