News for hackers. Stuff that matters.
Re:A/B testing gets misused to juice metrics in the short term by jerf on Tuesday July 12, 2022 @09:38AM (#32069722)
"As an experiment, I went through a list of holiday weekend sales, and opened all the sites. They all — all, 100% — interrupted my attempt to give them some money."

This is a good touchstone to use for "you've overoptimized your site, tone it back". I am also taken aback every time I'm on a site, I've got something in my shopping cart, I'm headed for the "check out" button, or I'm even on the checkout page, and some stupid interstitial pops up. Dude, I'm trying to enter my credit card information! Back off! Especially stupid for a "sign up for our newsletter" popup; we all know that unclicking the "yes, we can email you every 17 seconds from now until the heat death of the universe with valuable offers from 'our affiliates' which we define as 'anyone we share a species with'!" box on the checkout form is mandatory, and if we don't see it immediately we'd best go hunting for it. You've already default populating the checkbox to "yes" on this very screen, get out of the way!

Less unbelievably stupid, but related, is when I'm examining product X and just after I scroll down a bit to read more you pop up something related to... well... anything other than product X! I'm signalling interest in product X as hard as I can, and you've AB tested that this is a great time to jangle your keys over there instead? Your AB testing is stupid and can't possibly fail to be some stupid statistical fluke or other terrible error. What fisherman goes out on his boat, hooks a fish, and then rushes to throw another completely different lure out to the hooked fish and get them on that hook instead? This is another good touchstone for being "overoptimized".

Re:A/B testing gets misused to juice metrics in the short term by enlyth on Tuesday July 12, 2022 @10:02AM (#32070107)
Speaking of 'sign up to our newsletter', one of the latest dark patterns I've found that astounded me was adding a checkbox to the login form [0], where you'd normally expect the 'remember me' checkbox to be. You almost click it out of muscle memory if you don't read what it says.

[0] https://i.postimg.cc/HW89hs7r/Screenshot-2022-07-12-145957.p...

Re:A/B testing gets misused to juice metrics in the short term by eloisius on Tuesday July 12, 2022 @10:40AM (#32070670)
Email marketing in general blows my mind. Marketers typically have absolutely no respect for consent, and the costs are completely borne by the recipient. The whole industry depends on dark patterns, shady list sharing, and scraping your email to add it to their lists despite you having no relationship with them. I know it's not simple, and it's just my frustration speaking, but I don't understand how my mail host can't ban all Mailchimp et al IPs for me, or implement some standard such that it costs them a penny to send me an email.
Re:A/B testing gets misused to juice metrics in the short term by propogandist on Tuesday July 12, 2022 @11:33AM (#32071416)
beware that Meta/FB and Tiktok scripts are among those that siphon off email data even before a web form is submitted

https://arstechnica.com/information-technology/2022/05/some-...

Re:A/B testing gets misused to juice metrics in the short term by idrios on Tuesday July 12, 2022 @04:51PM (#32075519)
This is a good reminder to keep your hosts file updated to block at least some of these sites' attempts to take your data.

https://someonewhocares.org/hosts/

Re:A/B testing gets misused to juice metrics in the short term by fatnoah on Tuesday July 12, 2022 @11:48AM (#32071651)
> Email marketing in general blows my mind

My favorite thing is when the companies outsource the email marketing, so that it has absolutely zero relevance. I've been using the same online tax preparer for 10 years, and I've had exactly zero refunds, yet their emails during tax season always let me know that "my refund is waiting".

Re:A/B testing gets misused to juice metrics in the short term by brianwawok on Tuesday July 12, 2022 @10:25PM (#32078359)
The point of email marketing is to hit the right targets. If they overspray and hit 100000 wrong targets.., oh well, they lost 3 cents. So this overspray isn’t a sign that they are dumb, but they are effective.
Re:A/B testing gets misused to juice metrics in the short term by ryanmcbride on Tuesday July 12, 2022 @04:37PM (#32075344)
That's why the only email subscription service on my site is completely transparent and details exactly what we're storing, and it's impossible to click by accident. Everything I do for myself I try to build like a service I'd like to use myself. But the second managers or marketers are involved it's all out the window. I remember early in my career my boss had me add every email included in a TED booklet to their marketing email list. I told him that morality issues aside he could likely catch a fine for that, especially since the type of people listed in a TED booklet are likely more litigious than the average bear. Didn't care, wanted more eyes on the marketing.
Re:A/B testing gets misused to juice metrics in the short term by zx8080 on Tuesday July 12, 2022 @09:30PM (#32077982)
> That's why the only email subscription service on my site is completely transparent

Sorry, but honestly speaking even checking your profile does not reveal what your web site is. Probably you are marketing person from your words, but that's not clear.

Hope this feedback helps!

Re:A/B testing gets misused to juice metrics in the short term by ryanmcbride on Wednesday July 13, 2022 @01:09PM (#32085610)
That's intentional, I'm not trying to plug my website I was just talking about my approach in non-specific terms. I'm a dev.
Re:A/B testing gets misused to juice metrics in the short term by bbarnett on Tuesday July 12, 2022 @11:02AM (#32070994)
I'm not a big fan of ipv6, but fan or not, I bet if all spammy mailchimp type provider IP ranges were confiscated and freed, we'd be in ipv4 land for another 20 years.

And as a second thought, the way China amd Russia are going, maybe we should just reclaim all their ipv4 addresses, and just give each country 1 IP, they can proxy through it on their end.

Re:A/B testing gets misused to juice metrics in the short term by quickthrower2 on Tuesday July 12, 2022 @11:45AM (#32071608)
Then say goodbye to the internet and hello to a mesh of country-specific networks.

Finally, true decentralisation!

Re:A/B testing gets misused to juice metrics in the short term by bbarnett on Tuesday July 12, 2022 @12:40PM (#32072462)
? That's my point, as this is China already, and Russia is not far behind.
Re:A/B testing gets misused to juice metrics in the short term by jamiek88 on Tuesday July 12, 2022 @04:33PM (#32075281)
We already have that. It’s just we are in denial as a society about it.

Globalization is over. The post Berlin wall fall consensus undermined and ruined.

The last to realize, loses.

Re:A/B testing gets misused to juice metrics in the short term by account42 on Wednesday July 13, 2022 @05:09AM (#32080419)
The quickest way to achieve defeat is to convince yourself that you have already lost.
Re:A/B testing gets misused to juice metrics in the short term by GoblinSlayer on Wednesday July 13, 2022 @09:56AM (#32082774)
One address should be enough for everyone? Bill Gates just reached zen.
Re:A/B testing gets misused to juice metrics in the short term by bottled_poe on Tuesday July 12, 2022 @11:59AM (#32071822)
Mostly, it doesn’t even seem to matter whether you agree or not. Inevitably you end up receiving affiliate emails regardless.
Re:A/B testing gets misused to juice metrics in the short term by dspillett on Tuesday July 12, 2022 @01:09PM (#32072903)
This is why every company gets a different address for me. If junk starts coming in, that address is blocked, and I stop doing business with that company (if I haven't already).

This sometimes falls foul of spammers adding some random addresses of the form [email protected] or <commonname>@mycatchall.domain.tld into their lists, but that hasn't happened often enough to be a problem. That it isn't much of a problem surprises me a little, given how much <commonname>@domain.tld (no sub-domain) addresses are used this way. I have considered trying the pattern somename@<sub-domain-per-company>.domain.tld as an alternative if that becomes a problem, but before implementing that I need to change my email setup (doing that anyway soon as running Zimbra's OSS version is going to get more difficult next year) and maybe my DNS server of choice (if wildcard MX records are an issue, I've not looked into that).

Sometimes I get funny looks for addresses like this, especially as I usually work the company/other name in there somewhere. I had one website refuse to accept an address based on their name, which was a rad flag and I backed away from going any further into dealing with that organisation.

Re:A/B testing gets misused to juice metrics in the short term by aaaaaaaaaaab on Tuesday July 12, 2022 @01:51PM (#32073469)
You can just use [id]-[sha1 hmac]@domain.tld

The id could be anything, and the SHA1 HMAC takes 32 characters in base32 (which is an email-address-safe encoding). Then just configure your spamfilter to reject any address where the HMAC doesn’t check out.

Of course, the drawback is that you’ll need a computer to generate a new address… At which point you may as well store an explicit whitelist of valid addresses.

Re:A/B testing gets misused to juice metrics in the short term by remram on Friday July 15, 2022 @02:43PM (#32111450)
You can do that from a phone app though, or even a static webpage that loads on phones. It doesn't have to be 32 characters either, putting ~4 characters is probably secure enough, unless a ton of people start using this exact scheme.

This is a great idea, I might implement it. The difficult part is hooking the validation in the mail server I guess.

Re:A/B testing gets misused to juice metrics in the short term by dreamcompiler on Tuesday July 12, 2022 @06:18PM (#32076431)
I typically do it with

<emanynapmoc>@mydomain.tld

Spelling the company's name backwards makes it easy to match to a company for use by my own spam filter without setting off their pattern detectors.

Re:A/B testing gets misused to juice metrics in the short term by 3dGrabber on Tuesday July 12, 2022 @03:56PM (#32074863)
> If junk starts coming in, that address is blocked, and I stop doing business with that company (if I haven't already).

AND CALL them, if possible: “I’ve received marketing emails from your company recently, how is this possible, I’ve never signed up, yaddayadda… “

Generate some cost on their side.

Re:A/B testing gets misused to juice metrics in the short term by account42 on Wednesday July 13, 2022 @05:14AM (#32080445)
Support calls will probably cost you more than the company unless you value your time very little - most companis don't post numbers answered by anyone paid highly enough for this to have an impact. Instead, call them out publicly or mail/call a someone in a leadership position if you can find contact information.
Re:A/B testing gets misused to juice metrics in the short term by hahajk on Tuesday July 12, 2022 @08:14PM (#32077490)
You don't have to use the company name in the email address, just use a unique email. If you start getting spam on a certain email address, search your email archives to see what company was associated with that address to link it back to them.
Re:A/B testing gets misused to juice metrics in the short term by dwighttk on Tuesday July 12, 2022 @12:50PM (#32072625)
Yeah my mortgage company sent me a letter saying to opt out of affiliate marketing Emails or snail mail I have to send them a letter requesting it. This was 4 months after they bought my mortgage so the most the letter woulda done is stop them after selling my info for 4 months.
Re:A/B testing gets misused to juice metrics in the short term by Spooky23 on Tuesday July 12, 2022 @01:51PM (#32073462)
I had a coworker who sent those letters as a side hustle. He had a few different ones and would send the letters certified mail. Companies are very poor at compliance, and certain violations allow you to sue the company.
Re:A/B testing gets misused to juice metrics in the short term by dwighttk on Tuesday July 19, 2022 @07:06AM (#32149915)
Everything in that except the unspoken end point where I end up with money sounds awful
Re:A/B testing gets misused to juice metrics in the short term by bombcar on Tuesday July 12, 2022 @10:31AM (#32070516)
"Remember me in your newsletter list" is the next one. Send me money!
Re:A/B testing gets misused to juice metrics in the short term by cardamomo on Tuesday July 12, 2022 @10:17AM (#32070319)
Yes! I just noticed this for the first time yesterday and thought, "I hope this isn't another terrible trend in dark patterns."
Re:A/B testing gets misused to juice metrics in the short term by dwighttk on Tuesday July 12, 2022 @12:48PM (#32072587)
Heh. I’ve stopped clicking remember me boxes because they never work.
Re:A/B testing gets misused to juice metrics in the short term by aroccoli on Tuesday July 12, 2022 @10:22AM (#32070388)
Yeah, I booked a flight on WizzAir two days ago, and this felt like a low blow even from WizzAir.
Re:A/B testing gets misused to juice metrics in the short term by _puk on Tuesday July 12, 2022 @11:49AM (#32071663)
Predicting the flight will be cancelled in 3.. 2.. 1.. But then the newsletter will haunt you for far far longer..

Not a burnt WizzAir customer at all! /s

Re:A/B testing gets misused to juice metrics in the short term by enlyth on Tuesday July 12, 2022 @01:01PM (#32072794)
Happened to my dad on Sunday, and the only replacement flight they would offer is for this Friday, what a complete joke air travel is in 2022.
Re:A/B testing gets misused to juice metrics in the short term by nebusoft on Tuesday July 12, 2022 @02:23PM (#32073862)
I mean you're complaining about an ultra-low cost airline. why would you expect it to have a good customer experience?
Re:A/B testing gets misused to juice metrics in the short term by enlyth on Tuesday July 12, 2022 @03:43PM (#32074723)
Yes I am. They also overbooked seven people on that flight, delayed it for hours, and then completely cancelled. There were people in wheelchairs left stranded at the airport after waiting there all day. This is just plain incompetence.
Re:A/B testing gets misused to juice metrics in the short term by CodesInChaos on Tuesday July 12, 2022 @02:27PM (#32073908)
Re:A/B testing gets misused to juice metrics in the short term by geysersam on Tuesday July 12, 2022 @06:15PM (#32076388)
I knew Atlassian hates their users. But that much??
Re:A/B testing gets misused to juice metrics in the short term by Akronymus on Tuesday July 12, 2022 @10:39AM (#32070660)
Thats one of the things I REALLY dislike about GoG lately. It tries really hard to bait you into signing up for the newsletter when buying stuff.
Re:A/B testing gets misused to juice metrics in the short term by SoftTalker on Tuesday July 12, 2022 @10:44AM (#32070736)
I never check "remember me" so maybe that's good for me?
Re:A/B testing gets misused to juice metrics in the short term by nemo44x on Tuesday July 12, 2022 @10:53AM (#32070872)
A lot of this happens because different managers have different metrics/KPIs they are optimizing around and they all find "good places" to do things to help meet their goals. The secondary effects aren't considered. One managers quest for outperforming their goals comes at the expense of another managers goals.
Re:A/B testing gets misused to juice metrics in the short term by d23 on Tuesday July 12, 2022 @11:01AM (#32070981)
There was a point a few years ago where you could not see a single piece of user-generated content above the fold on the reddit home page. A bunch of teams had jockeyed for having their little carousels and banners put on top, and of course, metrics were always cited.

I left a screenshot in slack and it ended up causing a couple of teams to have to roll back their widgets, but it always baffled me that we were able to focus so much on the individual trees of metric optimization that we would miss the forest to that extent.

Re:A/B testing gets misused to juice metrics in the short term by nemo44x on Tuesday July 12, 2022 @11:32AM (#32071413)
> but it always baffled me that we were able to focus so much on the individual trees of metric optimization that we would miss the forest to that extent

Always look to the decision maker's incentives and you'll almost always discover why things are the way they are. And often, to your point, there's an aspect of tunnel vision associated with it because considering the bigger picture is difficult as a company grows and becomes more complex and creates friction in achieving goals.

Ultimately, this is the purpose of senior leadership. But the Peter Principal really begins to kick in at that level and the truth is, many senior leaders are in over their head and are unable to materialize the broader strategy and understand how their various units are affecting it. So we end up with crappy products.

Re:A/B testing gets misused to juice metrics in the short term by the_gipsy on Tuesday July 12, 2022 @02:20PM (#32073825)
Gervais principle says that senior leaders work for themselves, not the company, up to the point of working against the company.
Re:A/B testing gets misused to juice metrics in the short term by nemo44x on Wednesday July 13, 2022 @12:05AM (#32078937)
That was an interesting rabbit hole I hadn’t been down before. Interesting!
Re:A/B testing gets misused to juice metrics in the short term by bluGill on Tuesday July 12, 2022 @12:38PM (#32072449)
The best metric is the end of year bonus, tided to yearly total company financial results, but that only gets measured once per year. I can measure many things on every transaction, but how they in total work out to my end of year bonus and paycheck are much harder to see.

Of course if my bonus is some small KPI I can optimize that at the expense of overall performance.

Re:A/B testing gets misused to juice metrics in the short term by saagarjha on Tuesday July 12, 2022 @05:52PM (#32076172)
It seems like every company goes through some version of this. At Twitter there was a channel called #ios-six-bars or something like that that started when an engineer posted a screenshot of the home timeline with six bars of things on it, all from jockeying teams trying to grab a spot on that page: Home, Spaces, all the new features just had to be thrown in the face of somebody who probably just wanted to read some tweets. Discussions were had, product cohesion was brought up, then things went quiet for a bit. Until someone posted a new screenshot a couple months later with seven bars on the screen.
Re:A/B testing gets misused to juice metrics in the short term by spookthesunset on Tuesday July 12, 2022 @11:41AM (#32071536)
This is why any good org will make sure to observe all important KPI’s while doing an A/B test. If your “email signup” KPI went to the moon but tanked your “bought shit” metric… you should probably roll back.
Re:A/B testing gets misused to juice metrics in the short term by travisjungroth on Tuesday July 12, 2022 @03:35PM (#32074645)
It's really easy for this to be noise from false negatives. On an A/A test with five guardrail metrics and a threshold of p>0.05, you'll get a false negative 22.6% of the time.
Re:A/B testing gets misused to juice metrics in the short term by DrewADesign on Tuesday July 12, 2022 @05:52PM (#32076168)
Having experience design expertise at the executive level can mitigate this. If nobody is advocating for good user experiences, nobody is advocating for the usefulness of your online product as a whole, and it shows.
Re:A/B testing gets misused to juice metrics in the short term by retcon on Tuesday July 12, 2022 @11:39AM (#32071512)
The primary not secondary effect of random sampling is noise.
Re:A/B testing gets misused to juice metrics in the short term by CSMastermind on Tuesday July 12, 2022 @10:19AM (#32070347)
Recently I got an account on a developer tool (Checkly) because the company I joined uses it. I then got 5 different emails from them in a 48 hour period.

Like I'm sure many users sign up then drop out of your funnel but I'm part of an organization that's a paying customer. I'm already going to use your stuff. What possible business benefit could there be to you spamming me? If anything you're risking the inverse - it made me want to migrate away from the tool.

Re:A/B testing gets misused to juice metrics in the short term by chairmanwow1 on Tuesday July 12, 2022 @10:26AM (#32070449)
Sounds like they are optimizing for a KPI on time to full integration. Someone else paid for it, now they want to make sure that you are actually using it.

Still absurd, but I know this is a problem friends of mine have had.

Re:A/B testing gets misused to juice metrics in the short term by bombcar on Tuesday July 12, 2022 @10:34AM (#32070568)
The "here's how to use the product you paid for" emails that trickle out over a few weeks or a month is vastly different than "throw you into every single marketing email bucket we have" but companies seem to lean toward the latter.
Re:A/B testing gets misused to juice metrics in the short term by mattgreenrocks on Tuesday July 12, 2022 @10:24AM (#32070420)
The 'cost' of email is borne by the recipient, mostly.
Re:A/B testing gets misused to juice metrics in the short term by tnolet on Tuesday July 12, 2022 @11:02AM (#32071004)
Hey CSMastermind, I'm founder at Checkly and I got a ping we were mentioned. We do send out some "getting started" messages on autopilot. We also did a product launch Thursday and then our regular changelog on Monday. That probably was overwhelming. If you could email me on tim -at- checklyhq -dot- com I will track down if we hit the the spam cannon too hard.
Re:A/B testing gets misused to juice metrics in the short term by kaoD on Tuesday July 12, 2022 @11:33AM (#32071430)
I know this is well-intentioned and not an automated message, but I find it ironic that you managed to get an additional message to him over here.
Re:A/B testing gets misused to juice metrics in the short term by tpoacher on Tuesday July 12, 2022 @01:07PM (#32072880)
one asking them to establish contact for communication to continue, no less. xD

I can't help be reminded of those "if you'd like to unsubscribe call us so we can harass you with offers even more" you see in some company terms and conditions.

Re:A/B testing gets misused to juice metrics in the short term by musicale on Tuesday July 12, 2022 @06:54PM (#32076830)
> if we hit the the spam cannon too hard

Maybe consider, oh I don't know, not deploying a spam cannon in the first place?

The good news is that this sort of thing is enough to trigger the spam filter in my email program so I'll never see it.

Companies have this astonishing arrogance assuming they should be able to interrupt actual humans with robot messages from [email protected] - typically marketing messages that cannot actually be replied to and do not qualify as respectful communication.

Re:A/B testing gets misused to juice metrics in the short term by geoduck14 on Tuesday July 12, 2022 @08:34PM (#32077626)
>Maybe consider, oh I don't know, not deploying a spam cannon in the first place?

I'm not at all related to the people you are talking to, but I am a dev who has accidentally built spam cannons before. It is surprisingly easy to spam people - you need to deliberately build in checks - and put them in the right places to prevent spam cannons.

Re:A/B testing gets misused to juice metrics in the short term by wccrawford on Tuesday July 12, 2022 @11:11AM (#32071111)
I don't use your tool and I'm unrelated to CSMastermind, but I can tell you that you absolutely hit the "spam cannon" too hard. 5 messages in 48 hours? That's absolutely ridiculous. They could all have been 1 message if it was important that they get that info, and it could have been 0 messages if it wasn't.
Re:A/B testing gets misused to juice metrics in the short term by tnolet on Tuesday July 12, 2022 @11:18AM (#32071216)
You are correct. I will figure out two things.

1. Why our fancy expensive mailing/marketing tool (Intercom) does not spread these messages in a relaxed fashion (it should)

2. If of those five messages, maybe two were the obligatory "confirm your email" and standard "Welcome to Checkly, this is what we do"

TL;DR we should not spam.

Re:A/B testing gets misused to juice metrics in the short term by tnolet on Tuesday July 12, 2022 @11:51AM (#32071706)
We checked our Intercom. We spammed. One user got six emails in a 5 day span. There were some separate initiatives going on. We didn't check the settings and current outgoing mail. We will change it.

BTW: all of this was done without any bad intent. It's 100% us being stupid and not coordinating and being diligent enough.

Re:A/B testing gets misused to juice metrics in the short term by sonofhans on Tuesday July 12, 2022 @05:36PM (#32075998)
This is such an impressive series of responses. I know some folks are flipping you shit here, but I’ve seen a lot of people try to “engage” with customers, in HN and elsewhere, and I think you’ve done it really nicely. You sound like a human; you admit mistakes; you follow up. Good on you.
Re:A/B testing gets misused to juice metrics in the short term by tpoacher on Tuesday July 12, 2022 @01:10PM (#32072922)
Send a seventh to apologize! /s

(on a serious note, good on you for taking action though. +1)

Re:A/B testing gets misused to juice metrics in the short term by stalfosknight on Tuesday July 12, 2022 @12:27PM (#32072277)
You only get one chance to make a first impression.
Re:A/B testing gets misused to juice metrics in the short term by geoduck14 on Tuesday July 12, 2022 @08:35PM (#32077638)
>You only get one chance to make a first impression.

Well, this person had 6 first impressions.

Re:A/B testing gets misused to juice metrics in the short term by oxfordmale on Tuesday July 12, 2022 @03:43PM (#32074722)
tnolet, if I get five emails from a company with 48 hours, I will set up a spam rule for them. If I really need to get that much help to get started, your UX design is likely not very good.
Re:A/B testing gets misused to juice metrics in the short term by tnolet on Tuesday July 12, 2022 @04:04PM (#32074949)
You are correct. I’m the same. Our emails are even pretty good. Our UX is pretty damn good. Somehow we dropped the ball here.
Re:A/B testing gets misused to juice metrics in the short term by kthejoker2 on Wednesday July 13, 2022 @09:17AM (#32082226)
When you fix it, write up a post mortem so others can lesrn from your mistake!

Since it's distressingly common.

Re:A/B testing gets misused to juice metrics in the short term by joe_the_user on Tuesday July 12, 2022 @01:52PM (#32073479)
Let me propose a different possibility.

Suppose the site isn't concerned about the sale very much at all?

Suppose the thing that the site uses to reel people in, is a good deal that isn't very profitable to the site but what the site then tries to sell is a very profitable near-scam/ripoff. Scaring off half the ordinary customers becomes worth it to get even 10% of the customers buying the scam.

What seems like "poor optimization" can easily optimization for something and could be seen as "the scammification of the web".

Re:A/B testing gets misused to juice metrics in the short term by bsuvc on Tuesday July 12, 2022 @02:30PM (#32073947)
Exactly this.

Many here are focusing on a single interaction. While the outcome of that single interaction is negative to the company, the aggregate outcome must be positive somehow, perhaps in the way you said, but it doesn't even have to be a scam or ripoff. Some products just have a higher margin and/or customer LTV.

As an individual, it is annoying, but the company is focusing only on the macro effect when it does something like this.

Re:A/B testing gets misused to juice metrics in the short term by Jenk on Tuesday July 12, 2022 @10:03AM (#32070130)
> I'm signalling interest in product X as hard as I can, and you've AB tested that this is a great time to jangle your keys over there instead?

If I may... I have seen data from a big retailer that shows any user that doesn't immediately purchase an item, is actually not that interested in the product on the screen. If a customer is going to buy something, they will do it promptly. Anyone else is just browsing.

YMMV, grain of salt, context dependent, etc, etc.

Re:A/B testing gets misused to juice metrics in the short term by jerf on Tuesday July 12, 2022 @10:08AM (#32070177)
In this case, what I'm referring to is:

    1. Clicked on page.
    2. Took maybe 10 seconds to take in what is "above the fold".
    3. Scrolled down to see what else there is.
    4. BAM! Popup triggered by scrolling down.
While I understand what you're getting at, they do not yet have the info to know that I'm browsing or whatever. They were so excited about their stupid popup that they didn't even get that far.

I will say, generally, when I'm to the point that I'm entering credit card info, I've put up with it, but I have been chased off of sites by this use case before. Especially if that popup also crosses with some other popup and now I'm chasing down the tiny little 6pt light-grey-on-white little "x"s to click away the popups in the right order.

Actually, let me add that to my touchstone list. OF COURSE hiding the dismissal icon for the popup increases "engagement" with the popup. You don't even need to run a test for that, because what other result could it have? "We shrank the close icon, moved it to the lower right corner where nobody expects it, and made sure to kill the constrast even harder, and customers dismissed it 2.5 seconds more quickly on average"? Of course that's not possible. But... that's the wrong question! And AB testing is really good at answering the question you're asking, it has no mechanism in and of itself to see whether you're asking the right question. If you're getting down to this you've overoptimized.

Re:A/B testing gets misused to juice metrics in the short term by Akronymus on Tuesday July 12, 2022 @10:42AM (#32070702)
Or a popup that triggers for moving the mouse towards the top bar. I constantly highlight text for reading purposes (A habit I have) of course that moves the mouse. Not a reason to annoy me with that shit.
Re:A/B testing gets misused to juice metrics in the short term by Jenk on Wednesday July 13, 2022 @06:11AM (#32080737)
They know you clicked on a page that is for a given (set of?) product(s). They know you stayed on that page for longer than a moment, then scrolled down.

They may also have data that >90% of customers who make a purchase choose/click in <10 seconds, and never scroll. Or may even take a different path entirely or similar.

They could interpret all of that as you inspected the product(s) on the page, didn't choose/click one, and _now_ is the best time to prompt you with suggestions to maintain engagement in >75% of instances.

You could also be one of the lucky few customers in a sacrificial control group unknowingly testing their new shitty popup to measure its impact on sales.

Whilst I can't say for certain that any of this is your case, I absolutely guarantee there are many retail sites that do this - I've worked at several now - and this is bread-and-butter material to data teams working with customer engagement.

Re:A/B testing gets misused to juice metrics in the short term by dpe82 on Tuesday July 12, 2022 @10:54AM (#32070885)
In fact, you want the dismiss button to be easily discovered and used. Dismissals are an important signal about the quality of the content; just as important as clicks. When you make the button impossible to use you rob yourself of that signal while simultaneously making click data far less reliable.
Re:A/B testing gets misused to juice metrics in the short term by bbarnett on Tuesday July 12, 2022 @11:16AM (#32071187)
Unless you're a manager, not getting the results desired.

Then the problem is people, and "they're just clicking close out of reflex!".

Cue hiding this, and results you wanted appear! Success! Raise! Promotion! Or, maybe more funding, due to signs of greater engagement.

Re:A/B testing gets misused to juice metrics in the short term by dpe82 on Tuesday July 12, 2022 @05:05PM (#32075676)
Sure; until a company that actually understands this stuff comes along and eats that company's lunch. In the long run, reality eventually wins.
Re:A/B testing gets misused to juice metrics in the short term by sharemywin on Tuesday July 12, 2022 @10:15AM (#32070293)
The number to figure out is how much time do you wait to interrupt. Also wonder if it's person dependant. Some people aren't impulsive buyers.
Re:A/B testing gets misused to juice metrics in the short term by InCityDreams on Wednesday July 13, 2022 @08:23AM (#32081653)
>I have seen data from a big retailer that shows any user that doesn't immediately purchase an item, is actually not that interested in the product on the screen.

Fuck that. Unless it's an emergency (in which case I'll go to a shop), anything i purchase online is carefully considered, sometimes over several weeks. My revolving user-agent, vpn etc may give the illusion I'm not interested....but i am indeed, just browsing...

Re:A/B testing gets misused to juice metrics in the short term by axus on Tuesday July 12, 2022 @11:34AM (#32071433)
Could the popup be a punishment for reading the fine print?
Re:A/B testing gets misused to juice metrics in the short term by oxfordmale on Tuesday July 12, 2022 @01:17PM (#32073004)
I love to sign-up to news letters and get a discount. Of course I am giving you my spam account I set up for this exact purpose.
Re:A/B testing gets misused to juice metrics in the short term by naravara on Tuesday July 12, 2022 @10:28AM (#32070475)
Honestly instead of a cookie law I wish GDPR has imposed a rule that required all those stupid interstitial pop-ups to conform to a standard that could be easily blocked by the browser. I mean they are asking for emails, which is a massive and totally unnecessary proliferation of personally identifiable information.

I hate them so much. It makes it feel like so much more of a chore to try to do research or look for things online. I'd honestly prefer 56k page-load speeds if the pages were free of this garbage.

Re:A/B testing gets misused to juice metrics in the short term by jdlshore on Tuesday July 12, 2022 @11:25AM (#32071306)
Pet peeve: GDPR is not the cookie law, and is fact a very sensible collection of restrictions on how companies can collect and process personal data. The annoying banners you see are against the spirit of the GDPR, and quite possibly against the letter of it, too.
Re:A/B testing gets misused to juice metrics in the short term by m463 on Tuesday July 12, 2022 @10:24PM (#32078351)
Amazon does this in a horrible way.

1) I open a product in a tab. I click "add to cart" and a "related products" sidebar slides in. I close the tab in annoyance.

However, some items exhibit a similar pattern, EXCEPT...

2) I open a product in a tab. I click "add to cart" and a stupid extended warranty sidebar slides in. I close the tab in annoyance.

The difference?

Item #1 gets added to my cart

Item #2 doesn't make it to the shopping cart.

Amazon just silently deleted my purchase.

I actually don't know when it dawned on me that this happened, but amazon lost money on me because I didn't buy certain things.

Re:A/B testing gets misused to juice metrics in the short term by tshaddox on Tuesday July 12, 2022 @12:54PM (#32072687)
The thing is, you haven’t really shown that these sites aren’t successfully optimizing for conversions. Couldn’t it very well be the case that UI which annoys some high-intent users by interrupting them or adding steps to the checkout process also increases overall conversions?
Re:A/B testing gets misused to juice metrics in the short term by jl6 on Tuesday July 12, 2022 @01:03PM (#32072816)
True, A could be “annoy users” and B could be “don’t annoy users”, and A could perform better overall, but in this framework you might be missing C which is “annoy users except those already deep in the funnel”.
Re:A/B testing gets misused to juice metrics in the short term by tshaddox on Tuesday July 12, 2022 @01:31PM (#32073223)
My point is more that there can be two groups of users with mutually exclusive desires, and it can be practical to choose to satisfy one group over the other based on your particular goals. There's not always some monotonically increasing function where you can over time satisfy a higher and higher portion of your users.
Re:A/B testing gets misused to juice metrics in the short term by sharemywin on Tuesday July 12, 2022 @10:12AM (#32070241)
Most people are just browsing so it's optimizing for that when it should be optimizing for sales
Re:A/B testing gets misused to juice metrics in the short term by EGreg on Tuesday July 12, 2022 @10:45AM (#32070752)
Maybe your money is no longer the most important thing for them at that moment? Given that you’ll probably proceed with the purchase anyway, they could be making more overall from the crowd which also signs up for updates.
Re:A/B testing gets misused to juice metrics in the short term by tracerbulletx on Tuesday July 12, 2022 @05:43PM (#32076069)
I would really warn you against thinking your intuitions are going to be a good sign for whether or not something is a good retail decision.
Re:A/B testing gets misused to juice metrics in the short term by legalcorrection on Tuesday July 12, 2022 @11:49AM (#32071678)
Indeed, quantitative data without qualitative understanding is useless. You can't understand data without understanding mechanisms, because there's an infinite number of possible confounding factors that you can only dismiss through your qualitative understanding of the dynamic you are measuring.
Re:A/B testing gets misused to juice metrics in the short term by lifeplusplus on Thursday July 14, 2022 @12:58AM (#32092192)
What you describe could actually be an artifact of "flaw of averages" (an article was posted here just few days ago, TLDR: air force discovered that nobody fit the body dimensions of an average pilot and therefore created adjustable seats for jets). If promotion banners are just segmented to general millions of people then that's exactly what will happen. But not sure if better segmenting is even possible with data available.