The pitfalls of A/B testing in social networks

Re:The pitfalls of A/B testing in social networks by dalbasal on Monday October 16, 2017 @11:45AM (#15484015)

Many moons ago, OKC had a sort of clever & irreverent blog for stuff like this. It was a great read. There were some classics like "never pay for online dating[1]" and "The case for older women."[2] The first made a convincing case that the cheesy business model (see below for irony) of most dating sites doomed them to suckness. The second made a hilariously quantified (and also convincing) appeal for changing your age preference settings, while actually talking through real social norms and culture.

They managed to deal with touchy dating sterotypes, race, sex. Reading it, you could tell that they were genuinely smart (as opposed to just sophisticated) in their data/analytical mindset. You also got a feel for how the author thinks and where that mindset translates into making OKC.

Anyway, this reads like a linkedin article. Safe, vanilla company blog post. Big contrast. I wonder if the OKC guys writing that old blog 10 years ago new how unique their position was.

[1] Got taken down after they got bought by the target of the post. Accessible here: http://static.izs.me/why-you-should-never-pay-for-online-dat...

[2]https://theblog.okcupid.com/the-case-for-an-older-woman-99d8...

[ Parent | on HN ]

Re:The pitfalls of A/B testing in social networks by jdk on Monday October 16, 2017 @01:06PM (#15484809)

The cofounder of OKC and author of these posts for a long time wrote a book in the vein of the blog: https://www.amazon.com/Dataclysm-Identity-What-Online-Offlin...

[ Parent | on HN ]

Re:The pitfalls of A/B testing in social networks by dalbasal on Monday October 16, 2017 @01:12PM (#15484863)

Nice. Ordered

[ Parent | on HN ]

Re:The pitfalls of A/B testing in social networks by AznHisoka on Monday October 16, 2017 @12:42PM (#15484577)

I read somewhere that they had just 1 guy doing all the data analysis and after they got acquired he/she just was too busy with other work. Thus those type of posts just disappeared.

[ Parent | on HN ]

Re:The pitfalls of A/B testing in social networks by shostack on Monday October 16, 2017 @05:20PM (#15486724)

This is actually a hard thing to manage at some companies. Writers often don't have access or knowledge of the data, and engineers or data scientists often have bigger fish to fry beyond some content marketing explorations.

If anyone has found a good way to find that balance and create really solid in-depth data-driven content for there org, I'd love to learn more.

[ Parent | on HN ]

Re:The pitfalls of A/B testing in social networks by dalbasal on Monday October 16, 2017 @01:17PM (#15484908)

Makes sense. A founder is more likely to feel comfortable enough to take that tone, pick fights and take some risks.

Still, I imagine there's an underlying shift in company character that being expressed. The old blog felt like it was interesting to the writer.

This is in no way a condemnation of the post. I hope I haven't insulted the author.

[ Parent | on HN ]

Re:The pitfalls of A/B testing in social networks by dredmorbius on Monday October 16, 2017 @05:24PM (#15486745)

Discontinuing future work is one thing.

Censoring past published works is quite another.

[ Parent | on HN ]

Re:The pitfalls of A/B testing in social networks by WorldMaker on Monday October 16, 2017 @04:51PM (#15486544)

Semi-related aside, several of the original OKC founders are now at Keybase, and I feel like part of that blog culture you recall fondly is part of why Keybase articles get such high regards here at HN, beyond just the coolness of the tech they are working on.

[ Parent | on HN ]

Re:The pitfalls of A/B testing in social networks by vacri on Monday October 16, 2017 @10:14PM (#15488147)

For [2], the analysis is a bit off. For the mix/max age settings, there's not all that much difference between men and women for max age - for example a 30yo has a max partner age of 36 regardless of gender, and a 45yo has a max partner age of 50 regardless of gender. The difference in settings is in the min setting - and neither men nor women have the same range below as above (men quite a lot more, women a moderate amount less). Men never have an equal settings range above and below, and women don't hit an equal settings range until their 40s.

When it comes to actual messaging activity, both graphs show a 'belly' of preferring to message younger partners (except near the origin); it's just that the men's one is considerably more pronounced. Both genders are sending a lot of messages to people younger than their own minimum settings, too.

In general I liked the OKC data articles, but this particular one always seemed to me like they were forcing a story onto the data.

[ Parent | on HN ]