A/B testing gets misused to juice metrics in the short term

Re:A/B testing gets misused to juice metrics in the short term by wccrawford on Tuesday July 12, 2022 @11:11AM (#32071111)

I don't use your tool and I'm unrelated to CSMastermind, but I can tell you that you absolutely hit the "spam cannon" too hard. 5 messages in 48 hours? That's absolutely ridiculous. They could all have been 1 message if it was important that they get that info, and it could have been 0 messages if it wasn't.

[ Parent | on HN ]

Re:A/B testing gets misused to juice metrics in the short term by tnolet on Tuesday July 12, 2022 @11:18AM (#32071216)

You are correct. I will figure out two things.

1. Why our fancy expensive mailing/marketing tool (Intercom) does not spread these messages in a relaxed fashion (it should)

2. If of those five messages, maybe two were the obligatory "confirm your email" and standard "Welcome to Checkly, this is what we do"

TL;DR we should not spam.

[ Parent | on HN ]

Re:A/B testing gets misused to juice metrics in the short term by tnolet on Tuesday July 12, 2022 @11:51AM (#32071706)

We checked our Intercom. We spammed. One user got six emails in a 5 day span. There were some separate initiatives going on. We didn't check the settings and current outgoing mail. We will change it.

BTW: all of this was done without any bad intent. It's 100% us being stupid and not coordinating and being diligent enough.

[ Parent | on HN ]

Re:A/B testing gets misused to juice metrics in the short term by sonofhans on Tuesday July 12, 2022 @05:36PM (#32075998)

This is such an impressive series of responses. I know some folks are flipping you shit here, but I’ve seen a lot of people try to “engage” with customers, in HN and elsewhere, and I think you’ve done it really nicely. You sound like a human; you admit mistakes; you follow up. Good on you.

[ Parent | on HN ]

Re:A/B testing gets misused to juice metrics in the short term by tpoacher on Tuesday July 12, 2022 @01:10PM (#32072922)

Send a seventh to apologize! /s

(on a serious note, good on you for taking action though. +1)

[ Parent | on HN ]

Re:A/B testing gets misused to juice metrics in the short term by stalfosknight on Tuesday July 12, 2022 @12:27PM (#32072277)

You only get one chance to make a first impression.

[ Parent | on HN ]

Re:A/B testing gets misused to juice metrics in the short term by geoduck14 on Tuesday July 12, 2022 @08:35PM (#32077638)

>You only get one chance to make a first impression.

Well, this person had 6 first impressions.

[ Parent | on HN ]