News for hackers. Stuff that matters.
Re:Design tip: Never use black (2012) by gwern on Sunday October 20, 2013 @10:55PM (#6582982)
> My guess would be that the typeface on your site is so small (you're running that one now still?) that the lighter font might have made it harder to read.

Well, except that the bigger fonts are performing noticeably worse, and it's a pretty linear relationship. You can see it in the current simplified-model estimate of -0.002, but it jumps out when plotted: http://i.imgur.com/SKw3ScN.png The 100% group performs best, and it just gets worse, even though you wouldn't think, say, 105% would be much different from 100%.

> Couldn't it be that changing all four variables at once will make for a big boost in conversion, but changing one at a time runs into the limitations of the others? For example, increasing font-size by itself will lead to taller paragraphs and more black exposed.

Yes, that's entirely possible. Actually, it's probable that the fitness landscape looks like that. There's no reason to expect features to be independent and design so easily optimizable.

The problem is, it's hard to explore a high-dimensional landscape. Factorial designs, where you test every possible combination and estimate interactions, let you do this, but things can increase, well, factorially. 2x2 isn't too bad, but let's take just your 3 variables which can be tweaked by CSS: if I use just 4 differing shades of black, 5 font sizes (like the one I'm running now), and 5 differing line lengths (something like `max-width:`), then that gives me 4x5x5=100 different variants to run, and each one will need something like 1000 page views to start to become accurate, so it would take quite a while to accumulate 100,000 page views. (To put this in perspective, getting to #1 on the HN front page is worth ~20,000 page views.)

And that's just with 3 changes each with relatively few possible choices. You can see the problem. If you are really worried about interactions and decide to do a full factorial design to explore them, you can quickly get into infeasible sample sizes.

Re:Design tip: Never use black (2012) by ianstormtaylor on Sunday October 20, 2013 @11:44PM (#6583195)
Right, which is why I don't recommend designing by A/B testing single-variable changes in the first place. Instead of assuming that your site does better with extremely long lines of tiny type, even thought that goes against pretty much every typographer's recommendation, I'd assume that you started at a small local maximum if the results really do suggest you are best optimized.

                   _
                  / \
                 /   \    
    you?        /     \
     _         /       \
  __/ \_______/         \_______________

Another thing that came to mind: read time could actually be working against you, how do you know people are leaving because they can't read the type vs. they are leaving because it was so easy to read they finished earlier ;)
Re:Design tip: Never use black (2012) by gwern on Monday October 21, 2013 @10:32AM (#6585774)
> Instead of assuming that your site does better with extremely long lines of tiny type, even thought that goes against pretty much every typographer's recommendation

It's not really an assumption though, I've already looked at long lines with the `max-width` variable, and the very wide/long versions performed pretty well; see http://www.gwern.net/AB%20testing#max-width & http://www.gwern.net/AB%20testing#max-width-redux

Re:Design tip: Never use black (2012) by ianstormtaylor on Monday October 21, 2013 @01:23PM (#6586888)
That was a single-variable test, right? I just talked about that...
Re:Design tip: Never use black (2012) by pygy_ on Monday October 21, 2013 @09:31AM (#6585338)
His definition of conversion is unusual (compared to what we usually see on HN).

His articles are long, and take more than 40 seconds to read.

Re:Design tip: Never use black (2012) by gwern on Monday October 21, 2013 @10:27AM (#6585731)
Right. It's impossible to genuinely read any page on my site in <40 seconds, so I can treat an increase in >40seconders as an increase in reading time.