Re:Feature Flags: Theory vs. Reality
by SketchySeaBeast on Monday July 10, 2023 @02:43PM
(#36671080)
> "Test was successful so it's rolling out to all users, minus a 0.5% holdback population for the next 2 years"
Man, I couldn't imagine being a user in such a situation. "Oh, I guess I'm just not getting the better functionality?" Even worse if I were a paying customer.
It’s actually usually the paying customers asking via support to be added to the holdback, improved experience or no.
This is more true for larger flags that substantially change the experience and may not implement niche or edge-case functionality. Obviously you want to avoid these kinds of tests if possible but it’s not always possible.
Users should not be allowed to select their treatments; it defeats randomization, which is what allows causal inference.
Re:Feature Flags: Theory vs. Reality
by mandelbrotwurst on Monday July 10, 2023 @06:50PM
(#36674081)
Sure, they'll be more predictive that way, and simultaneously it's valuable to not piss off your customers.
In that case I would take them out of the experiment and impute the censored data.
Re:Feature Flags: Theory vs. Reality
by SketchySeaBeast on Monday July 10, 2023 @03:21PM
(#36671582)
Ah, that makes way more sense.
Re:Feature Flags: Theory vs. Reality
by Brian_K_White on Monday July 10, 2023 @02:58PM
(#36671287)
You are probably the lucky elite who got to keep the funnctionality you wanted.
e.g. "Test was successful so it's rolling out to all users, minus a 0.5% holdback population for the next 2 years"
This then forces the team to maintain the two paths for the long-term, ensuring the team might get re-orged / re-prioritize their projects sometime a year later making the cleanup really hard to eventually enforce.