Well, except that the bigger fonts are performing noticeably worse, and it's a pretty linear relationship. You can see it in the current simplified-model estimate of -0.002, but it jumps out when plotted: http://i.imgur.com/SKw3ScN.png The 100% group performs best, and it just gets worse, even though you wouldn't think, say, 105% would be much different from 100%.
> Couldn't it be that changing all four variables at once will make for a big boost in conversion, but changing one at a time runs into the limitations of the others? For example, increasing font-size by itself will lead to taller paragraphs and more black exposed.
Yes, that's entirely possible. Actually, it's probable that the fitness landscape looks like that. There's no reason to expect features to be independent and design so easily optimizable.
The problem is, it's hard to explore a high-dimensional landscape. Factorial designs, where you test every possible combination and estimate interactions, let you do this, but things can increase, well, factorially. 2x2 isn't too bad, but let's take just your 3 variables which can be tweaked by CSS: if I use just 4 differing shades of black, 5 font sizes (like the one I'm running now), and 5 differing line lengths (something like `max-width:`), then that gives me 4x5x5=100 different variants to run, and each one will need something like 1000 page views to start to become accurate, so it would take quite a while to accumulate 100,000 page views. (To put this in perspective, getting to #1 on the HN front page is worth ~20,000 page views.)
And that's just with 3 changes each with relatively few possible choices. You can see the problem. If you are really worried about interactions and decide to do a full factorial design to explore them, you can quickly get into infeasible sample sizes.
_
/ \
/ \
you? / \
_ / \
__/ \_______/ \_______________
Another thing that came to mind: read time could actually be working against you, how do you know people are leaving because they can't read the type vs. they are leaving because it was so easy to read they finished earlier ;)It's not really an assumption though, I've already looked at long lines with the `max-width` variable, and the very wide/long versions performed pretty well; see http://www.gwern.net/AB%20testing#max-width & http://www.gwern.net/AB%20testing#max-width-redux
His articles are long, and take more than 40 seconds to read.
And if gwern feels adventurous (and I guess he'd like testing such kind of things), he may try something like [1] to color the text with a gradient, to allow faster reading speeds.
As in line height? Already tried that: http://www.gwern.net/AB%20testing#line-height
> [1] http://www.beelinereader.com/
I emailed them when they came out for a JS version which I could use. I guess they are still working on it because I haven't heard anything lately.
[1] http://www.readability.com/read?url=http://www.gwern.net/AB%...
But it can still help. A local optima is still higher than not being in the local optima.
I haven't looked at Readability a lot. For a while, it wasn't displaying gwern.net pages at all... and it's still messing up a lot. Apparently the Readability view starts halfway down the page! /hits the report-bug button yet again.
- pure black fonts are actually bad for conversion, and conversion gets worse the more pure black that is exposed (ie. the size of the type).
- small font sizes are also bad for conversion.
- long line lengths are also bad for conversion.
- tall paragraphs (in number of lines) are also bad for conversion.
Now I'd imagine (purely on design instincts) that the best possible reading experience (which I'm assuming also means more reading time) would be to have off-black-colored type, at a size around 16px, with a line length of 70-80 characters, and an average paragraph height of around 4-5.
Couldn't it be that changing all four variables at once will make for a big boost in conversion, but changing one at a time runs into the limitations of the others? For example, increasing font-size by itself will lead to taller paragraphs and more black exposed.