Could there be such a thing as "meta-malicious design?" Bad consequences aren't executed directly, but conditions are put in place that cause the emergence of bad consequences.
I suspect, if AI ever does away with Homo sapiens it will be by convincing us not to breed by changing our incentive structures.
I think it'll be by just sitting back and waiting a few decades :)
The sci-fi tinfoil hat interpretation of this: It's already here, or at least its cronies and enablers, and the plan has already been put into effect.
— John Stuart Mill, 1867
For instance, I was in charge of design for a mortgage lender's web site. I begged my manager to let me try (for lulz) an A/B test replacing the photo of the woman call centre operator next to the quote form with a photo of a Labrador retriever wearing a headset (you knew he was a professional because he also wore glasses and a necktie). The dog form consistently outperformed the woman by 19% but we had to roll it back after a couple of weeks because.. well.. dog.
There is not some painstaking language optimization designed to trick users. But there is not a filter for clarity either, so the A/B testing naturally favors "dark patterns" - it's not malicious design, it's just unintended consequences, and gives the impression that Amazon et al are deliberately hostile when they are really just ignorant