Re:The pitfalls of A/B testing in social networks
by jerednel on Monday October 16, 2017 @05:52PM
(#15486926)
> But even that can be tricky, where one group of people might click fewer ads but are higher quality
This is why I always pull performance metrics in addition to impressions/clicks when running A/B tests. Not being in the media/creative department means I often don't see the creatives i'm asked to analyze the performance of so there are occasionally different creatives that have a stronger resonance with people with a higher propensity to convert. This is itself a learning and could lead to using that sort of creative more for acquisition focused campaigns, rotating it out of the upper funnel ad rotation and coming up with a new upper funnel creative to test for the purposes of building large audience pools.
> This is especially troublesome when you can't tell how many friends they invite through word of mouth.
Could you provide an incentive (gold/credit/whatever) for each new active user referred which could feed into the evaluation criteria of the test as far as the "value" that a particular conversion brought with it? Then I suppose it turns more into a LTV study.
This is why I always pull performance metrics in addition to impressions/clicks when running A/B tests. Not being in the media/creative department means I often don't see the creatives i'm asked to analyze the performance of so there are occasionally different creatives that have a stronger resonance with people with a higher propensity to convert. This is itself a learning and could lead to using that sort of creative more for acquisition focused campaigns, rotating it out of the upper funnel ad rotation and coming up with a new upper funnel creative to test for the purposes of building large audience pools.
> This is especially troublesome when you can't tell how many friends they invite through word of mouth.
Could you provide an incentive (gold/credit/whatever) for each new active user referred which could feed into the evaluation criteria of the test as far as the "value" that a particular conversion brought with it? Then I suppose it turns more into a LTV study.