videophile

Bad example. The whole discussion is about reporting percentages of percentages.

Look, it's pretty clear that hackers have left the building and let marketroids rush in. Feel free to use your ill-defined metric. If you have the least bit of intellectual curiosity, you'd let everyone know what you'd report when the value goes from 0 to a non-zero value. That is all.

Values reported as percentages should generally be values between 0 and 100. So when the real conversion rate goes from 0.002% to 10%, the reported number should not be 499900%.

I expect hackers to use metrics that are at least defined over the range they are supposed to be used. Imagine that the conversion rate goes from 0% to 1%. I report 1% improvement, the value I measured. You? (1-0)/0 = what percent?

>take note of the "wtf is this guy talking about" downvotes you've received on this thread

I don't rely on opinion-polling to determine who is right, esp. when it comes to questions related to science.

>conversation with anyone else on the planet

On your planet, a move from 0.0001% to 0.001% would be reported as "an improvement of 1000%" No thanks.

> It's about metrics that matter and revenue.

Yes, we established long ago that it's about marketing-speak versus math.

So, since you have so many upmods, perhaps you can tell me what happens in your world when the conversion rate goes from 0% to 1%? :-)

An equally valid question to ask is: "How many users are bouncing away from that website after landing on the first page?" That raw rate improved by 5.2%. When you quote percentages of percentages, the rates vary widely depending on your point of view -- in the original example, an optimist might report a 20% improvement, looking at the users who reached point B, and a pessimist might report a 10% improvement, looking at the users who left without reaching point B. This is only partly why it's a bad idea to quote percentages of percentages -- they are dependent on the baseline. There are at least two baselines to choose from (successful conversions vs bounces). Also, percents of percents hide the task difficulty, which varies depending on the starting point (going from 1% to 2% is probably not hard, going from 99 to 100% is probably very very hard).

But the other reason why it's a bad idea to quote percentages of percentages is that there is no indication whether the claimed improvements are actually statistically significant. I didn't see any data in the article to indicate they are, and what little I know about conversions from my own observations is that they vary by at least 10% depending on totally random factors.

>Which is a better description: "I doubled my conversion rate" or "I increased my conversion rate by 1%?" I'd go with the former.

I see your point and I'd use it if I needed to flatter myself. But I'm sticking to the latter. And I'd be sure to mention the starting baseline as well.

>what is the standard deviation in that example...You need sample sizes to even attempt to do the math.

First, you're asking the wrong person to supply the missing data. The onus is on the OP to justify their stats, not on the person calling BS on the misuse of statistics.

Given that the original article is missing the critical numbers and presenting the rest in the most self-serving manner possible, the only data anyone has to go by is their own experience with conversion rates. We don't even know what "conversion" means for this website because they don't define it, but that justifies making reasonable assumptions (e.g. users who land on page A end up on page B, for a website in some industry, involving some side-effects to get from A to B that are left undefined by the OP). In my experience, the behavior of users on a web site is quite fickle and dependent on the surrounding advertising campaigns, as well as the weather, the time of day, and the timing of holidays. And maybe a billion other confounding factors. So, is the 5% improvement seen here significant? We don't know, there's no evidence to assume it is, and that's precisely my point.

The improvement in the conversion rate is (29.6 - 24.4) = 5.2%.

Reporting "a percent of a percent" is downright misleading because it is entirely dependent on the starting baseline and also because conversion rates have high natural variance. Imagine an improvement from 2% conversion (for a truly awful site) to 2.6% (for a site just as awful). That "30% improvement" just isn't. If they were intellectually honest and called it a "0.6% improvement," anyone would be able to see that the claimed improvement is well within stddev.

There is some "New Math" being invoked here. If the conversion rate goes from 24.4% to 29.6%, it's disingenuous to claim a 21.2% improvement.

I stopped at the first misuse of statistics, but by that point, they had taken out informative links from their web site and turned it into a landing page. Frankly, a 5% change in conversion rate is in the noise compared to the effect of targeted advertising (and I have no faith that these sloppy reporters actually did a properly controlled experiment).

Edit: Downmodded for asking for real math over marketing math?

Ok, I'm not a PC-nazi but I think someone needs to call the article out:

This is generic claptrap ("use help"?! he, and it's pretty clear it's a he, could have at least looked at how hard emacs help is to use for beginners) interspersed with sexist pictures and commentary. It is offputting to many people, and I'd like to see my favorite editor associated with higher caliber tutorials.