News for hackers. Stuff that matters.

A comment on Claude Code's binary reveals silent A/B tests on core features. Parent comment

Re:Claude Code's binary reveals silent A/B tests on core fea... by londons_explore on Saturday March 14, 2026 @12:32PM (#47378344)
A big tech company has ~10k experiments running at once. Some engineers will be kicking off a few experiments every day. Some will be minor things like font sizes or wording of buttons, whilst others will be entirely new features or changes in rules.

Focus groups have their place, but cannot collect nearly the same scale of information.

[ Parent | on HN ]
Re:Claude Code's binary reveals silent A/B tests on core fea... by rkomorn on Saturday March 14, 2026 @12:39PM (#47378427)
I think a lot of people (myself included) would just like to not be constantly part of some sort of revenue optimization effort.

I don't care, at all, about the "scale of information" for the company's sake.

[ Parent | on HN ]
Re:Claude Code's binary reveals silent A/B tests on core fea... by londons_explore on Saturday March 14, 2026 @04:38PM (#47380916)
Often the experiments are not for revenue - many of them will be optimizing user experience metrics - ie. Load time or user dropoff rate.

They are clearly good for both user satisfaction and the companies bottom line.

[ Parent | on HN ]
Re:Claude Code's binary reveals silent A/B tests on core fea... by jjj123 on Saturday March 14, 2026 @05:35PM (#47381465)
As someone who works in these orgs, only a small fraction are about user experience metrics. 90+% are extracting more short term value with unknown second order effects on usability.
[ Parent | on HN ]
Re:Claude Code's binary reveals silent A/B tests on core fea... by franktankbank on Saturday March 14, 2026 @03:48PM (#47380422)
If you have 10k experiments running then you are probably p-hacking.
[ Parent | on HN ]
Re:Claude Code's binary reveals silent A/B tests on core fea... by wavefunction on Monday March 16, 2026 @11:14AM (#47400108)
Big tech companies are not serving their "users" but advertisers, it's a common mistake.
[ Parent | on HN ]