A small tech team investigated fraud on our platform and developed a system that was pretty robust at detecting and potentially shutting it down. But literally nobody was interested - even the people advertising don't want to know.
The people spending money are typically networks, media buyers, ad agencies, etc, far removed from the actual brand.
There are so many parties who want a slice of the brand's cash that they are all long past caring about whether the ad is being viewed by a human or not.
IIRC our system wasn't even fancy ML - just finding known user IDs with identical timestamps over many simultaneous clicks / impressions on really diverse publishers.
I assure you the big brands CFO's take a huge amount of interest in what is spent on advertising.
Other dept;s like IT have to justify every penny, but Sales and Marketing not sooo much
I've seen it not just in the phenomenon you describe, but also in layoffs and restructurings.
Engineering , R&D, Manufacturing, Tech, Testing : "cost centers". Ever tightening budgets.
MBA, Beancouting 101
That said, companies like P&G, Airbnb, and Uber, which are oft-cited as examples of digital not being worth it, fail to understand their own brand recognition and organic power, built through prior marketing efforts, as key to their current standing.
Sure, TODAY, it doesn’t have the impact they’d like it to have but the investments PRIOR were key to ensuring their success.
I worked next desk to people running a small ad agency. Because we shared office (and I found them an intern), I got a very good look at how they're working. What I've seen can be summarized as: people who have zero clue or interest in statistics writing "reports", with "graphs" they don't even understand beyond "pointing up = good", proving positive ROI to customers - who also have zero clue or interest in understanding the numbers in the report, and not enough visibility into the whole funnel to independently check attribution. Both the agency and the clients were engaging in a shared and completely unjustified fiction of positive advertising return - and as long as both sides were happy, the money kept flowing.
I've been long since suspecting a lot of advertising on-line looks like that. Every now and then, I see evidence in favor. Like that good ol' Optimizely debacle, where it turned out Optimizely was structurally optimized to help people make invalid A/B tests, that erred on the side of concluding the interventions were working[0]. And sure, big brands with some superstar ad teams probably do this right. But I think there's enough slack in most businesses that advertising spend can get quite far detached from actual ROI without anyone noticing (and with plenty of people happily riding the gravy train).
--
The large ad firms deliver a very good product but it usually isn't cheap.
I've personally dealt mostly with the other end - the individuals and small agencies - and what I saw revealed total lack of necessary competences for the reports to be corresponding to reality. Perhaps it's understandable - after all, people who have the required knowledge likely end up working for big advertisers, or in entirely different fields. But small business owners don't pick these big advertising companies either.
The question isn't are the scientists good, the question is if the vendor is honest.
In a huge number of cases these tests are run by people that don't have the statistical background to property run and understand these tests, in the remaining cases there is almost never follow through to ensure that the results of the test have persisted after the experiment.
People will say "oh this test shows 10% improvement, and this one does too! and this one! and this one!" But then you should see a nearly 50% improvement just because of ad spend and you almost never do. Nobody ever check that all the numbers add up, they just want the numbers that someone reported to feel like they are sound enough to hold up to scrutiny but scrutiny is never applied.
It seems like it would be important to measure this standing and turn down spend once it is reached. I can see how there's a lack of incentive to help large advertising spenders understand this.
I worked at an ad company. It was an absolutely standard metric to eg geo-fence ads out of a state or two for 3 months to demonstrate the impact of ads. This isn't easily externally visible, but tests like this are standard practice.
Particularly in the app install space, which is sketchy as hell once you stop buying from the top handful of vendors, buyers should be auditing by a couple million in annual spend. To get to $150m without looking hard at big chunks of their spend is just plain arrogance and/or incompetence.
The technical aspect aside, there are a lot of "soft" factors that help: regular communication of easily consumable numbers/graphs/metrics to the client, calling out inconsistencies, etc. In other words, this is far from a fire-and-forget exercise; this can be an intense monitoring and client-engagement exercise.
The regular client engagement sometimes helps you in pinpointing cause-effect in the observations too. For ex company X might decide to advertise less of product Y in a certain state Z because of new laws there. But the team you're interacting with might not be aware of this change, or might not be cognizant of its potential impact on an advt. campaign. Regular dialogue helps here since you might observe a change in sales trends, and bring it up in a meeting - and the client team might be able to then rationalize the change. This is healthy for both parties.
~Chomsky to Andrew Marr.
Because everyone is already acting like they know it works, so the only way that experiment can change things is in a way that's bad for the person in question. In that situation, they should (from a local, selfish perspective) be resisting even if they're awfully sure it does work (and perhaps even if they're right!).
Given that, I don't think the behavior has already given us an answer.
The goal was to demonstrate an impact of in-store sales from internet advertising. They did the first studies in about 2008-10, and have continued running these studies ever since.
They even built a tool so that advertisers can run these studies, and get a sense for the incremental impact of their ad.
Google have a similar (less full-featured) system.
Really, it's the rest of the ecosystem that has much of the fraud, and I think that a lot of people in the industry are aware of this.
So the goal was to advertise advertising. A more scientific goal would be to see if there was an increase of in-store sales from internet advertising. Instead they were looking to design experiments that would show a positive effect, with the goal of giving advertisers a dashboard so they could run those experiments themselves.
I think you are misinterpreting what "do the experiments themselves" implies.
Certainly there is a JOIN with Google's or FB's data, but it can still happen entirely in a hosted sandbox with no data sharing necessary.
We've run Google's Local Campaigns, which are supposed to drive brick and mortar store visits, which they measure using GPS and "other signals". I checked with one of our most remote and least frequented stores, to see whether they could see any of the 100 daily visits that Google claimed to have generated, and they couldn't see a thing (avg daily visits were around 80 before during and after the campaign)
And the social stigma in science of being someone who tries to take down, discredit, or disprove your colleagues/superiors/ competitors theories is pretty substantial: IME there has historically almost been a taboo against attacking our speaking negatively about publications and your own sciences + faculties practices.
The saying of sciences advancing one funeral at a time doesn't exist because they're all such great skeptics and falsifiers, and current scientific practice is heavily biased towards positive findings and contains general publication biases.
indeed there's actually a LOT of common ground with advertising self-interest, since a lot of publication in science is effectively just advertising your brand...
This is a dangerous misconception to be spreading. It is not at all related to your other claims, namely:
> Sampling can skew results. Scientists are people too. We all have flaws.
These are of course correct statements. But they do not influence truth. They may well influence people's understanding of what the truth is, but that's different.
Same for a lighter, there are many factors involved, and playing with lighters that way was a party trick where I grew up.
We use truth as a concept every day as an approximation, but the universe is not bound to follow. Very unlikely things happen all the time.
And yes, when we put things that way it brings a lot of possible issues with sampling, processing, and measuring (and who’s doing the measuring). Some of these things would be harder to control for in a study of the effectiveness of advertising.
Any large company could invest in some experimentation, whether their marketing directors want it or not. It makes sense that at least a few do but just don’t publish the results
And then using data from a one system that went down and nothing happened as a proof that systems reliability doesn’t matter at all, and it’s huge scam by engineers.
Or they have run the experiment and the results haven't lines up with their own personal biases so they were discarded.
Because advertising in some form certainly works. If you can determine that approach "A" that everybody is doing is actually a waste of money but approach "B" is effective, then you can develop services around approach "B" and market them based on these findings.
It also happens to be the sole form of revenue all of the largest tech firms enjoy.
No wonder there hasn't been a real audit.
Also, all those marketing execs buying all the ads would be out of a job too, so they aren't speak up either.
I don't think this is true unless you qualify it a bit. Apple, Amazon, Microsoft, don't make most of their money on ads. Even Google has other revenue streams.
How long do they survive in their current form relying solely on them?
The people that would have the power to run this experiment have their entire careers depending on things staying as-is. Running the experiment carries a significant risk of exposing that the advertising operations they're responsible for provide much less ROI than they pretend it does.
The unwillingness of anyone to run such an experiment is already an answer. Why wouldn't someone jump at an opportunity to prove the thing/service they provide actually works, unless they were unsure about it themselves?