News for hackers. Stuff that matters.
Re:How Netflix uses Druid for realtime insights by devmunchies on Monday March 16, 2020 @01:12AM (#22590201)
> During software updates, we enable the new version for a subset of users and ... compare how the new version is performing vs the previous version. Any regression in the metrics gives us a signal to abort the update and revert

How do you account for the possibility that the update only performs badly because it’s different than what users are used to, but would actually be an improvement in the long run?

Re:How Netflix uses Druid for realtime insights by Twirrim on Monday March 16, 2020 @11:46AM (#22593654)
> How do you account for the possibility that the update only performs badly because it’s different than what users are used to, but would actually be an improvement in the long run?

That's assuming that the update contains a UI/UX change. A lot of updates that will roll out won't include that, they'll be fixing or optimising things.

Re:How Netflix uses Druid for realtime insights by catnzhat on Monday March 16, 2020 @02:00AM (#22590408)
You acknowledge that fact and if you're making a severe change you either run a longer test or you break up the large change into manageable chunks (Google Maps is a great example of this).

For something that is effectively binary and can't be changed incrementally the only way is to extend the evaluation window or make a call with limited information.

Re:How Netflix uses Druid for realtime insights by 2rsf on Monday March 16, 2020 @08:46AM (#22592194)
regressions due to "users are used to" usually goes away after a while, you'll see a temporary dip in the results.

If you are convinced it's better for the long tun then what's the point in measuring ?

Re:How Netflix uses Druid for realtime insights by Joof on Monday March 16, 2020 @01:39AM (#22590313)
Have you worked at a large company before? You don't!