The key here is that it is a cohort analysis.

So while your analysis is correct and the cancellation rate will likely rise over time, the key thing to check when comparing a pre-change cohort to a post-change cohort to test for improvement is that the cancellation rates at two weeks post-signup are significantly different. Obviously its the most ideal if you can run the tests in parallel to reduce the risk of selection bias. However, sometimes that is not always feasible, and with a result like 85%+ improvement is not really necessary to do so in order to assume that the control was beaten.

I don't really understand. Isn't the "cohort analysis" part of it how the data about the gyms that cancelled was extracted? That shouldn't affect how you compare the results once you've made changes. I'm also not sure I understand what you mean by "comparing a pre-change cohort to a post-change cohort" because we're not comparing the cohorts, we're comparing the cancellation rate among the whole list of customers.

Ah, the article is actually extremely unclear. What I took his conclusion to mean was that some cohort of gyms that had signed up post-change had lower cancellation rates than pre-change customers. If the analysis just took into account the overall rate across all customers is prone to all sorts of errors (including the one you mention).

My impression was that the pre-change cancellation rate was based on all customers over the lifetime of the product (they did calculate the 61 day average from this), whereas the post-change was only for the new customers in the last two months (the max it could be, and the only thing it makes sense to measure).

I think I see what you're saying (given an average cancellation time, only half the customers will have cancelled by that time), but I think it depends on how they calculated it.

For instance, they could know the average number of cancellations in a week, and see how that number has dropped.

I think this would work out, provided it doesn't matter too much whether the two actions they've taken (the two emails) occur right when the gym signs up or later on.

You're right that an average time to cancel of 61 days implies some may cancel at, say, 80 days. But some gym signed up 20 days before this cohort analysis began, and could have been the one to cancel had it not been for these changes.