News for hackers. Stuff that matters.
Re:Conservation of Intent: why A/B tests aren’t as effective... by citrablue on Tuesday July 03, 2018 @04:59PM (#17453576)
What's the alternative to A/B testing? Everywhere I've worked, design teams hate A/B testing, but suggest "trust us we know what we're doing" as the alternative.

A/B/n testing let's you explore a search space. There are issues that can come up from that, but IMO it's a better option than this Dilbert comic[0].

[0] http://dilbert.com/strip/2014-10-27

Re:Conservation of Intent: why A/B tests aren’t as effective... by swied on Tuesday July 03, 2018 @05:32PM (#17453765)
A/B testing is a great tool for incrementally improving upon a product design given the correct business strategy. Just don't let the limitations of A/B testing drive how you choose your business strategy. The strategy is ultimately about maximizing total profit, and not necessarily short term profit per user.
Re:Conservation of Intent: why A/B tests aren’t as effective... by citrablue on Tuesday July 03, 2018 @11:10PM (#17455042)
How do you create a discipline around making good decisions for business strategy? I've seen "highest paid person" win, I've seen "Most vocal" win, I've seen "5 versus 1" win.

The testing advocates I've worked with have an attitude of "I don't know what will work - that's why we're testing". I have not seem much of that attitude from most other individuals/teams involved in decision making -- they have preferred to say "this will work", and get angry when I say, "compared to what? how do you know?"

edit - fwiw, I totally agree with understanding the limitations of testing. You have to know what it can be good for. My argument is that it's actually better for long term strategy than most other decision making processes I have experienced, which usually boils down to "gut based".

Re:Conservation of Intent: why A/B tests aren’t as effective... by s-shellfish on Tuesday July 03, 2018 @07:21PM (#17454331)
Listen to people about their complaints and preferences as intellectual equals.
Re:Conservation of Intent: why A/B tests aren’t as effective... by citrablue on Tuesday July 03, 2018 @11:06PM (#17455037)
Sure, but not everyone's opinion is equally valid. I've worked with a number of design teams who confuse their opinion (an educated opinion, which I respect) as a fact. Facts beat opinions -- if a short-term optimization will be a long term loser, then prove it.

Unfortunately, it's not really possible to prove/disprove an opinion on what a design element will do.

Additionally, optimization testing allows you to try out different options. Going with someone's (educated) gut feel is a worse decision than exploring the option space.

Ideally, you'd test everything. In the real world, resource constraints apply, so at issue is how to establish an internal discipline where facts trump opinions, and opinions can be tested then either discarded or confirmed.

Re:Conservation of Intent: why A/B tests aren’t as effective... by s-shellfish on Tuesday July 03, 2018 @11:18PM (#17455070)
I mean the users of software. Instead of trying to predict what is the best design, create interfaces that make it easy to aggregate that data, and take that information at face value.

When it comes to design, there often aren't facts (one may even suggest facts don't exist when it comes to design). Study the history of art to contest that. Otherwise, listen to users like they are humans, and see whether it matches with the designers advice. Simple.

Re:Conservation of Intent: why A/B tests aren’t as effective... by Franciscouzo on Tuesday July 03, 2018 @11:09PM (#17455040)
I think a better approach might be using bayesian optimization, each variation would be a dimension, and the algorithm would find the (almost) optimal possibility instead of a/b testing, which finds the local maximus.