A/B testing gets misused to juice metrics in the short term

Re:A/B testing gets misused to juice metrics in the short term by tnolet on Tuesday July 12, 2022 @11:02AM (#32071004)

Hey CSMastermind, I'm founder at Checkly and I got a ping we were mentioned. We do send out some "getting started" messages on autopilot. We also did a product launch Thursday and then our regular changelog on Monday. That probably was overwhelming. If you could email me on tim -at- checklyhq -dot- com I will track down if we hit the the spam cannon too hard.

[ Parent | on HN ]

Re:A/B testing gets misused to juice metrics in the short term by kaoD on Tuesday July 12, 2022 @11:33AM (#32071430)

I know this is well-intentioned and not an automated message, but I find it ironic that you managed to get an additional message to him over here.

[ Parent | on HN ]

Re:A/B testing gets misused to juice metrics in the short term by tpoacher on Tuesday July 12, 2022 @01:07PM (#32072880)

one asking them to establish contact for communication to continue, no less. xD

I can't help be reminded of those "if you'd like to unsubscribe call us so we can harass you with offers even more" you see in some company terms and conditions.

[ Parent | on HN ]

Re:A/B testing gets misused to juice metrics in the short term by musicale on Tuesday July 12, 2022 @06:54PM (#32076830)

> if we hit the the spam cannon too hard

Maybe consider, oh I don't know, not deploying a spam cannon in the first place?

The good news is that this sort of thing is enough to trigger the spam filter in my email program so I'll never see it.

Companies have this astonishing arrogance assuming they should be able to interrupt actual humans with robot messages from [email protected] - typically marketing messages that cannot actually be replied to and do not qualify as respectful communication.

[ Parent | on HN ]

Re:A/B testing gets misused to juice metrics in the short term by geoduck14 on Tuesday July 12, 2022 @08:34PM (#32077626)

>Maybe consider, oh I don't know, not deploying a spam cannon in the first place?

I'm not at all related to the people you are talking to, but I am a dev who has accidentally built spam cannons before. It is surprisingly easy to spam people - you need to deliberately build in checks - and put them in the right places to prevent spam cannons.

[ Parent | on HN ]

Re:A/B testing gets misused to juice metrics in the short term by wccrawford on Tuesday July 12, 2022 @11:11AM (#32071111)

I don't use your tool and I'm unrelated to CSMastermind, but I can tell you that you absolutely hit the "spam cannon" too hard. 5 messages in 48 hours? That's absolutely ridiculous. They could all have been 1 message if it was important that they get that info, and it could have been 0 messages if it wasn't.

[ Parent | on HN ]

Re:A/B testing gets misused to juice metrics in the short term by tnolet on Tuesday July 12, 2022 @11:18AM (#32071216)

You are correct. I will figure out two things.

1. Why our fancy expensive mailing/marketing tool (Intercom) does not spread these messages in a relaxed fashion (it should)

2. If of those five messages, maybe two were the obligatory "confirm your email" and standard "Welcome to Checkly, this is what we do"

TL;DR we should not spam.

[ Parent | on HN ]

Re:A/B testing gets misused to juice metrics in the short term by tnolet on Tuesday July 12, 2022 @11:51AM (#32071706)

We checked our Intercom. We spammed. One user got six emails in a 5 day span. There were some separate initiatives going on. We didn't check the settings and current outgoing mail. We will change it.

BTW: all of this was done without any bad intent. It's 100% us being stupid and not coordinating and being diligent enough.

[ Parent | on HN ]

Re:A/B testing gets misused to juice metrics in the short term by sonofhans on Tuesday July 12, 2022 @05:36PM (#32075998)

This is such an impressive series of responses. I know some folks are flipping you shit here, but I’ve seen a lot of people try to “engage” with customers, in HN and elsewhere, and I think you’ve done it really nicely. You sound like a human; you admit mistakes; you follow up. Good on you.

[ Parent | on HN ]

Re:A/B testing gets misused to juice metrics in the short term by tpoacher on Tuesday July 12, 2022 @01:10PM (#32072922)

Send a seventh to apologize! /s

(on a serious note, good on you for taking action though. +1)

[ Parent | on HN ]

Re:A/B testing gets misused to juice metrics in the short term by stalfosknight on Tuesday July 12, 2022 @12:27PM (#32072277)

You only get one chance to make a first impression.

[ Parent | on HN ]

Re:A/B testing gets misused to juice metrics in the short term by geoduck14 on Tuesday July 12, 2022 @08:35PM (#32077638)

>You only get one chance to make a first impression.

Well, this person had 6 first impressions.

[ Parent | on HN ]

Re:A/B testing gets misused to juice metrics in the short term by oxfordmale on Tuesday July 12, 2022 @03:43PM (#32074722)

tnolet, if I get five emails from a company with 48 hours, I will set up a spam rule for them. If I really need to get that much help to get started, your UX design is likely not very good.

[ Parent | on HN ]

Re:A/B testing gets misused to juice metrics in the short term by tnolet on Tuesday July 12, 2022 @04:04PM (#32074949)

You are correct. I’m the same. Our emails are even pretty good. Our UX is pretty damn good. Somehow we dropped the ball here.

[ Parent | on HN ]

Re:A/B testing gets misused to juice metrics in the short term by kthejoker2 on Wednesday July 13, 2022 @09:17AM (#32082226)

When you fix it, write up a post mortem so others can lesrn from your mistake!

Since it's distressingly common.

[ Parent | on HN ]