S[it]: number of sessions originating on page [i] at time [t]
b0: base traffic for all pages
A[i]: base traffic for page [i]
X[it]: seasonal traffic for page [i] at time [t]
D[it]: dummy variable, 1 for after treatment 0 for control and before treatment
b3: effect size of treatment
e: noise/error term
Which gives a formula in the form of y = mx from which we can calculate the effect size b3 and the p value using linear regression.
The key assumption is that X[it] - E[X[i]] = X[jt] - E[X[j]] for large numbers of pages. I found this assumption does hold in practice. I'm not a statistician so forgive me if I've made any statistical errors but I think this analysis is correct.
I might write up a blog post at some point to explain it in more detail.
The key assumption is that X[it] - E[X[i]] = X[jt] - E[X[j]] for large numbers of pages. I found this assumption does hold in practice. I'm not a statistician so forgive me if I've made any statistical errors but I think this analysis is correct.
I might write up a blog post at some point to explain it in more detail.