There is also absolutely no will in the marketing departments I've seen to measure negative effects of advertising, only the positive.

Myself I have an adverse effect when I see an ad for a product. I will actively try to avoid it as I feel if you have to advertise, the competition probably has a superior product.

I completely agree. I feel this way for things that I think should not have to advertise. For instance, higher education. I saw a poster at Forest Hills 71 Avenue subway station for Queens College and now I feel like it is not a top class college.

I think my reaction would have been different of the ad was subtler though. Thoughts?

Ads for colleges in the subway aren’t meant for you, though. You probably already have a college degree and are well-educated. Those subway ads are for people who aren’t as well-educated, never went to college and are trying to fit it in between working, and wouldn’t get into a top-tier college.

Exactly. The top-tier college in the city probably has a subway stop with its name in it. Those ads are aimed at someone who doesn't have a college degree and/or has been contemplating doing something positive about their career. So part-time/night college, nursing school, etc. ads are there to nudge them and to suggest a specific option for them to consider.

I believe the thought was that if they were so prestigious they would not need to advertise at all. I have never felt this way about Universities, but I certainly have for defense attorneys.

Of course they advertise. They just do it in subtle ways called PR.

I used to do marketing for HE, and that always used to be the case - however the explosion in intakes has really skewed the market. Increase in tuition fees has also turned it into more of a transaction, so it's now less about selective prestige, and more about outcomes (which lends itself more to outward marketing than traditional reserved means).

There is, but will is different than ability - I've done work on brand perception studies, which look at both the positive and negative, as well as conversion data looking at online ads that work from a vanity metric standpoint (views/clicks) but don't do anything else.

A lot of brands will try and piece together negatives - if you want to help a marketer, fill out that NPS popup you see ;)

I do not understand why it’s hard to see at least the end result. Before switching on the ads and after switching on the ads.

I am sure a company as big as proctor and gamble has at least a team of people who are dedicated to gather the end to end metric. Then where’s the gap?

One big part of the problem is whether that team (if they exist at all, and that's a big IF) works for the marketing department and whether they have an inherent conflict of interest.

In my experience as a data scientist working along-side some marketing people, most of the performance measuring, if it was ever done, was total psuedo-science. It was almost universally done either by people aligned with the actual marketing or who actually worked on said marketing effort.

Not only is there the universal mantra of "Don't ever come to a finding which says that your team and effort fails to contribute value to the company", but lets say that you actually are a data scientist and you really are independent and you come to a conclusion that your multi-million dollar marketing campaign is doing sweet f all.

This will generally be despite 6 months of presentations from the marketing team about how great it was, and probably several bonuses/pay reviews.

And now you're not only fighting a civil war in your company, you're doing it against multiple people with millions of dollars in budget, who have been proclaiming their success, and who's entire career has been based upon appealing to peoples guts to give them high paying jobs and big budgets.

Good luck with that fight...

Rule 37 of professional analytics in big companies: be VERY careful before turning your analytical eye inwards into your own companies' hierarchy/practices...

I think we're reaching that crux point now - it's always been a black box of advertising online, but now teams are taking some knowledge in-house, there's increasing accountability, and thus cutting of budgets as they're clocking onto ROI (which a lot of channels can't show).

  I do not understand why it’s hard to see
  at least the end result.

The longer it takes for a change in input to produce a change in output, the greater the chance confounding factors will interfere, making your conclusions less convincing.

You moved your spending from billboards to print magazines, and 6 months later sales are higher. How do you tell if it was the ads, when there was also a model refresh, your competitor released a poorly reviewed product, and the economy has been kind to your target demographic?

As John Wanamaker supposedly said, "Half the money I spend on advertising is wasted; the trouble is I don't know which half"

Your decision to buy Nike sneakers might be influenced by Nike's sponsorship of Uruguay in the 1998 World Cup. Maybe you were a child back then, and thought their goalkeeper was just the coolest dude, ever.

See where the problem might be?

Recently I was buying some gym clothes and wanted to buy Nike. I stopped to think about why, and traced it back to the early 2000s when I got into golfing, and Tiger Woods was a big influence for me. At that time I bought all Nike clothing.

After realizing that I just need some tops and shorts to sweat in at home, I ended up buying the store brand label, saving myself ~80% in costs.

It's funny how the mind works.

Others have done a great job of highlighting some of the challenges, but I wanted to share my own thoughts.

Conducting an on/off test can be a viable approach in some cases assuming you can properly control for the numerous (understatement) variables that often make your data incredible dirty for such a test. That said, often the nature and number of the variables is such that you would not get meaningful results from an on/off test. This is quite often the case with large brands. As such, other forms of testing are often used individually or in conjunction with each other to get a better read on things.

In terms of the end-to-end data, from an online advertising standpoint you often are missing trackable data points around other brand touch points the person might have encountered before purchase, or often, even the end purchase itself. And that's just the start of the data headaches.