Excellent point, but it shouldn't impact an A/B test. Specifically, the experimental and control users should be equally drawn to look at the deactivation page.

However, this could give a false impression of improvement if you just look at the data before and after the launch. It's a good reason to also look at the total number of deactivations / visits to the deactivation page.