There is not some painstaking language optimization designed to trick users. But there is not a filter for clarity either, so the A/B testing naturally favors "dark patterns" - it's not malicious design, it's just unintended consequences, and gives the impression that Amazon et al are deliberately hostile when they are really just ignorant

Could there be such a thing as "meta-malicious design?" Bad consequences aren't executed directly, but conditions are put in place that cause the emergence of bad consequences.

I suspect, if AI ever does away with Homo sapiens it will be by convincing us not to breed by changing our incentive structures.

I think it'll be by just sitting back and waiting a few decades :)

The sci-fi tinfoil hat interpretation of this: It's already here, or at least its cronies and enablers, and the plan has already been put into effect.

— John Stuart Mill, 1867

For instance, I was in charge of design for a mortgage lender's web site. I begged my manager to let me try (for lulz) an A/B test replacing the photo of the woman call centre operator next to the quote form with a photo of a Labrador retriever wearing a headset (you knew he was a professional because he also wore glasses and a necktie). The dog form consistently outperformed the woman by 19% but we had to roll it back after a couple of weeks because.. well.. dog.

At any given time, a single engineer might have a couple of A/B tests running. That's the single biggest engineering change I have seen in the last 20 years.

A/B tests are now crazy simple to set up; just click a few buttons and decide how much traffic to send. The major con is that it leaves a lot of dead code behind when the best of N A/B test arms are launched; the (N-1) unsuccessful arms stick around as dead code.