News for hackers. Stuff that matters.
Re:Ad blocking is under attack by p0pcult on Thursday October 27, 2022 @09:19AM (#33357059)
There's a lot more collusion going on here as well. I work in the ad measurement/ad tech industry; it is rife with misaligned incentives.

Ad measurement companies have no incentive to accurately report the incremental sales (i.e., that which would not have happened, had the ad not aired) of the ad they are measuring. Why not? Because their clients, the marketing function at the advertiser, don't get their bonuses paid based on the accuracy of the measurement firm's work product. They get their bonuses (and ultimately salary) based on the success of their marketing efforts. If a company like mine reports that an ad on TikTok for Dran-o didn't sell any Dran-o that wouldn't have been sold anyway, Dran-o's marketers will just find a different ad measurement company.

Furthermore, as a platform, TikTok has to prove to potential advertisers that TikTok is a good platform on which to place ads. So TikTok also has no incentive for accurate measurement.

So, what does a company like mine do? Juice the numbers. Because the company leadership is judged not on accuracy of measurement, but profit.

It's a lot like the bond rating agencies that lead, in part, to the 2008 housing crisis, getting paid by the people whose homework they were grading.

This is an entirely true story, btw. Apparently if you advertise Dran-o to kids on TikTok, you get something north of a $2.00 ROI. Completely ridiculous.

In another anecdote, we measure advertising on YouTube. Google would only let us implement changes to our measurement software if they "improved" the software. (Again, I don't mean accuracy. Neither did Google.)

How do you solve this? Well, you can start by letting the CFO at the advertiser own the ad measurement responsibility. The CFO actually has incentive to understand if the marketers are just pissing away money. The shareholders do too.

Perhaps there's also room for an industry watchdog, paid for by advertising platforms and advertisers, that certified the practices of measurement companies?

Anyway, shareholders are getting fleeced. All so the Valley can rake in beaucoup bucks at the expense of shareholders. With the added benefit of a layer of surveillance apparatus suffusing the internet.

Fun times!

There's actually a lot of scholarship on how advertising's effects are a lot lower than the ad measurement and advertising industry thinks they are. (Go figure that the people who profit from advertising think it works.) Freakonomics Radio episodes 440 and 441 are a good place to start, for those who are interested.

Re:Ad blocking is under attack by jklinger410 on Thursday October 27, 2022 @09:38AM (#33357297)
You work in the ad measurement/ad tech industry, so this should be an easy question for you to answer.

When your solutions are "juicing the numbers" how are they addressing the miscorrelation between third party tracking tools for both impressions and clicks on the platforms, but also what would come through in analytics in the site or app?

Without once mentioning a pixel and how that all works, you claim that numbers are being inflated, how so? Are you talking about over-reporting, mostly caused by impression conversions?

If you are "getting a $2 ROI showing Dran-O ads to kids on TikTok" there have to be sales occurring in the first place in order for TikTok to claim credit for them.

Can you also go into more detail about what tracking solution for YouTube you needed to contact Google about? That is confusing to me.

I've been in the ad world for almost 8 years now. If a platform is over-reporting conversions I have other ways to audit that.

Re:Ad blocking is under attack by p0pcult on Thursday October 27, 2022 @09:56AM (#33357507)
<edit> I should point out that we were measuring brick & mortar sales, not clickthroughs. So no cross site tracking involved. Maybe that's the easiest way to clear things up for you? </edit>

If you look at a distribution or histogram of ROIs from our thousands of studies, you will see that it approximates a gaussian distribution...right up until the left hand side hits $1.00, where it looks like the distribution hit a brick wall.

Same thing for a distribution of incremental lifts, except that the brick wall is right above 0% lift.

It's a discontinuity that shouldn't exist, like the Polish high school exit exams. See if you can guess what the minimum score to pass* was:

https://imgur.com/4nnbyru

Except, unlike that distribution, there is virtually nothing to the left of the discontinuity at my company.

Without talking about the tech behind it, what I can say is that the fraud is not occurring on the impressions and tracking side. It's in the statistical tools that infer the incrementality of the purchasing behavior. Seriously, I urge you to listen to the Freakonomics episodes I mentioned.

*The minimum score was 30. This is pretty damning evidence that teachers were juicing the tests of kids who filed by one or two questions.

Re:Ad blocking is under attack by whimsicalism on Thursday October 27, 2022 @11:08AM (#33358681)
> we were measuring brick & mortar sales, not clickthroughs

I think this is probably where you get the room to juice the numbers like this. If you were tracking online conversions, people would easily be able to compare with an incrementality study run by the provider (say Google) and see that you are lying/wrong.

Re:Ad blocking is under attack by p0pcult on Thursday October 27, 2022 @11:34AM (#33359145)
Yes, and the fact that "incrementality" is inherently unobservable.
Re:Ad blocking is under attack by whimsicalism on Thursday October 27, 2022 @11:47AM (#33359360)
Disagree. Why do you say that?
Re:Ad blocking is under attack by p0pcult on Thursday October 27, 2022 @12:19PM (#33359876)
Replying here, because I cant reply to your comment several levels down.

I can see how you would have that perspective as a third-party analyst of conversion.

It's actually easy to do if you are the exchange/sell-side provider of the advertising space. In your logic, where the advertiser running the incrementality study would have won the auction, just randomly remove their bid from some won auctions, have the second highest bidder win, and then track that user as the "control" for conversion purposes vs. users you did display the ad to. Voilà!

That's not observation. That's statistical inference. I am referring, strictly, to direct, empiricial observation. With any kind of statistical inference, you get into a pissing match about assumptions, model quality, etc... Is the "control" a good stand-in for the behavior of the "test", etc.

Incrementality is able to be estimated, but unobservable. And therein lies the potential for fraud.

Re:Ad blocking is under attack by whimsicalism on Thursday October 27, 2022 @12:23PM (#33359944)
Sure, if by unobservable you mean in the same way that we can't truly "observe" the population average height of humans, even if we can probably get a pretty darn good estimate.

You can observe the results of the experiment and then make statistical inference about the "true" incrementality. You don't even have to perform inference necessarily, you can just provide the direct results of your experiment to the advertiser and let them do the inference.

But I do fail to see how a third-party could possibly infer this measure, given that they have no capacity to do a true incrementality experiment on the platform.

Re:Ad blocking is under attack by TeMPOraL on Thursday October 27, 2022 @01:49PM (#33361291)
Are you running the incrementality study, or is your provider (e.g. Google)? Because one of the points here is that providers also have an incentive to fudge the numbers - they're not in business of being accurate, they're in business of making it seem their services help you.

To riff off the average height example - you can get a pretty darn good estimate based on sampling and good statistical practice... unless the people doing the sampling have a reason to bias the measurement, or - if you're sampling on your own - people who control where you go and who you meet...

Re:Ad blocking is under attack by whimsicalism on Thursday October 27, 2022 @01:57PM (#33361392)
Oh that could definitely be the case, although I'll note that the larger actors have more to lose and are at more risk of being caught for engaging in fraud.

I think my usage of "you" was confusing because I work in sell-side advertising.

Re:Ad blocking is under attack by tomrod on Thursday October 27, 2022 @07:02PM (#33365193)
You nailed this. Tirole discusses this in his paper hierarchy on bueracracies. Your internal stakeholders aren't always aligned to the company, sometimes it is to a vendor.
Re:Ad blocking is under attack by p0pcult on Thursday October 27, 2022 @12:11PM (#33359741)
Incrementality, as defined in my original post, is the sales that would have not occurred, had the ad not occurred.

It is impossible to actually observe what would have happened, had there been no advertisement, because there in fact was an advertisement.

Thus, the estimation of incrementality must be inferred statistically.

Re:Ad blocking is under attack by whimsicalism on Thursday October 27, 2022 @12:14PM (#33359776)
> It is impossible to actually observe what would have happened, had there been no advertisement, because there in fact was an advertisement.

I can see how you would have that perspective as a third-party analyst of conversion.

It's actually easy to do if you are the exchange/sell-side provider of the advertising space. In your logic, where the advertiser requesting the incrementality study would have won the auction, just randomly remove their bid from some won auctions, have the second highest bidder win, and then track that user as the "control" for conversion purposes vs. users you did display the ad to. Voilà!

This is how Google in-house incrementality studies are run, for instance.

Re:Ad blocking is under attack by jklinger410 on Thursday October 27, 2022 @12:31PM (#33360094)
Thanks for the info. Doing brick and mortar sales attribution is a whole different ball-game.

Frankly the number of dots that have to be connected between different systems in order to make that stuff work is too difficult for me to trust the data. If I'm going to trust a black box, I'm not going to trust programmatic exchanges and attribution vendors. I'm going to trust platforms that have vertical integration at as many stages as possible.

So, Google basically.

Re:Ad blocking is under attack by dec0dedab0de on Thursday October 27, 2022 @09:34AM (#33357232)
Your point about the numbers being fudged makes intuitive sense, and almost feels obvious, however...

I definitely remember asking my mom to buy liquid plumber when I was about 13 because I had a slow drain in a bathroom she never used. I only realized that slow=clogged from the commercials I saw.

Re:Ad blocking is under attack by p0pcult on Thursday October 27, 2022 @09:38AM (#33357284)
That's fair, and reasonable. However, to a data scientist, this looks like an anecdote and not data.
Re:Ad blocking is under attack by dec0dedab0de on Thursday October 27, 2022 @02:10PM (#33361583)
Yes, it was definitely just a fun anecdote. Your example of advertising drano to kids made me laugh, and then I remembered being the reason we bought drain cleaner in my home, which made me laugh again, so I shared it.
Re:Ad blocking is under attack by p0pcult on Thursday October 27, 2022 @02:38PM (#33362018)
I definitely appreciated it, though that probably didn't come through in my comment. Thanks for sharing!
Re:Ad blocking is under attack by Operyl on Thursday October 27, 2022 @09:42AM (#33357351)
Make that n=2, probably one of the only useful things I learned from a commercial as a kid.
Re:Ad blocking is under attack by marcosdumay on Thursday October 27, 2022 @10:21AM (#33357846)
Hum... You are claiming the GP's claim that the reported numbers are higher than the real ones is false because you once on your lifetime got evidence that the real number is non-zero?
Re:Ad blocking is under attack by DangitBobby on Thursday October 27, 2022 @10:49AM (#33358372)
I think he's claiming that showing drano ads to kids could have a positive ROI based on lived experience. To be fair, while the parent post was compelling, it didn't have any "references" either.
Re:Ad blocking is under attack by marcosdumay on Thursday October 27, 2022 @11:54AM (#33359475)
The OP never claimed that showing ads to kids didn't have positive ROI.
Re:Ad blocking is under attack by DangitBobby on Saturday October 29, 2022 @09:12PM (#33389832)
You did not understand the original comment. They implied it without saying it in as many words. Here's how: they presented the fact that Drano apparently gets a $2 ROI showing ads on TikTok as evidence to back up their claim that the companies measuring the effectiveness of ads are making shit up. You can argue whether that's the intended communication, but clearly that's the communication I received (and probably not just me, hence the other person explaining that Drano ads for kids could have a positive ROI).

As a side note, I must say I feel this conversation took on a negative (and condescending?) tone.

Re:Ad blocking is under attack by TeMPOraL on Thursday October 27, 2022 @01:35PM (#33361078)
> Your point about the numbers being fudged makes intuitive sense, and almost feels obvious, however...

I can confirm that point on a small scale; it's something I repeatedly mentioned on HN for years. I used to work literally next desk to a small social media marketing agency, that spun out of the company I was employed by. I knew the people running that agency, I got to see how they work, even read and commented on the reports they sent to customers.

It was as GP described, with a twist - the people working in said agency weren't exactly good with numbers. They understood, however, the idea that a graph going up and to the right is good. Despite the numbers having little connection to reality, they wrote reports that highlighted how successful their campaigns were. Those reports were read by customers equally proficient in mathematics, and similarly good at tracking sales on their own side. Bottomline: money changed hands, both sides were happy, the whole thing was still completely untethered from reality.

Re:Ad blocking is under attack by tomrod on Thursday October 27, 2022 @05:12PM (#33364112)
Advertisements used to be that exactly, telling you about a product for a use case.

You probably didn't see that ad at the super bowl.

Re:Ad blocking is under attack by tomrod on Thursday October 27, 2022 @09:36AM (#33357261)
This is a fantastic series of points.Thanks for sharing them.
Re:Ad blocking is under attack by p0pcult on Thursday October 27, 2022 @09:40AM (#33357321)
For the economist in you:

There's a book on this topic that explicitly draws the parallels to the housing market meltdown, "Subprime Attention Crisis." There is also a professor at Columbia, Kinshuk Jerath, who has written explicitly about these misaligned incentives.

Re:Ad blocking is under attack by whimsicalism on Thursday October 27, 2022 @11:05AM (#33358643)
If you aren't the one supplying the ad space or the exchange (ie. Google or FB), then I don't believe you can actually measure incrementality on your end.
Re:Ad blocking is under attack by p0pcult on Thursday October 27, 2022 @12:15PM (#33359793)
You just need the data on who does (and by extension, doesn't) see the ad, as well as the shopping behavior of those same people. Both of which can be had by someone who isn't supplying the ad space or the exchange.
Re:Ad blocking is under attack by whimsicalism on Thursday October 27, 2022 @12:19PM (#33359863)
> who does (and by extension, doesn't) see the ad

Not sure how you get this data if you are not the supplier.

You just mentioned in another comment that the key data needed to measure incrementality is impossible to get, from your perspective. [0]

[0]: https://news.ycombinator.com/item?id=33359741

Re:Ad blocking is under attack by p0pcult on Thursday October 27, 2022 @12:25PM (#33359982)
No, you misquote me. I said that is is impossible to observe incrementality. It is rather trivial to estimate incrementality, statistically, but all statistical models produce estimates. Not observations.

How do you get the data if you're not a supplier? You contract for the supplier.