What if they follow through with it 6 months or a year later? Your a/b metrics won't catch long term effects. Maybe a change gets people more engaged short term, but drives them away long term.