Yet I'd like to add that I don't think that testing frameworks (at this point it would be misleading to call them strictly A/B testing) HAVE to only reflect short term gains. It's hard to come up with (proxy) metrics that hold that kind of short term optimization in check.
Just to pick one fairly obvious example for e-commerce: you can track returns/customer service contacts as a health metric or even explicitly assign a value to them to fold them into the primary metric you're optimizing.
These kinds of safeguards typically need longer recording periods, so it's potentially a lot of data science and engineering effort to build tools that can handle the long term data collection and analysis. But it's not impossible. It's rather something people love to pretend is not a problem.
Can you expand? I am an Ars reader, and I too find it frustrating compared to what it used to be. I wish there were more in-depth technical articles; I find it too light on details, written for a non-tech audience. I really, really wish it had solid technical content, since it's what got me reading it.
But I don't characterise it as click-bait (it seems clear) nor as especially political, except in so far as politics intersects technology and climate. In both those it seems balanced or erring towards freedom.
That's always going to throw a wrench into automation. So yes, it's hard work that requires real thinking with a clear focus on the users and how the users feel about the software. Software needs to care about users if it is dependent on users for it's business model!
A/B/n testing let's you explore a search space. There are issues that can come up from that, but IMO it's a better option than this Dilbert comic[0].
The testing advocates I've worked with have an attitude of "I don't know what will work - that's why we're testing". I have not seem much of that attitude from most other individuals/teams involved in decision making -- they have preferred to say "this will work", and get angry when I say, "compared to what? how do you know?"
edit - fwiw, I totally agree with understanding the limitations of testing. You have to know what it can be good for. My argument is that it's actually better for long term strategy than most other decision making processes I have experienced, which usually boils down to "gut based".
Unfortunately, it's not really possible to prove/disprove an opinion on what a design element will do.
Additionally, optimization testing allows you to try out different options. Going with someone's (educated) gut feel is a worse decision than exploring the option space.
Ideally, you'd test everything. In the real world, resource constraints apply, so at issue is how to establish an internal discipline where facts trump opinions, and opinions can be tested then either discarded or confirmed.
When it comes to design, there often aren't facts (one may even suggest facts don't exist when it comes to design). Study the history of art to contest that. Otherwise, listen to users like they are humans, and see whether it matches with the designers advice. Simple.
What are they testing for in their metrics? Is it just click rates? Well that’s a patently losing strategy in the long term.
What if whether or not you came back over time is what was being measured? Would that break some cardinal rule that all A/B tests must be done within a short time period?
Researchers has been doing A/B testing for centuries. Till now they’ve just called it having a control variable and an independent variable. Experimental design is a big deal, and something that is taught to every graduate student researcher, but I don’t see the tech blogs giving experimental design enough focus — at least not in their click-bait headlines. Sometimes people don’t design good experiments and then blame the methods.
Mountains can be moved with randomized control trials.
Something as basic as mod(hash(experiment_group_id or layer id, userid), 1000) would give you a user stable experiment distribution that you can analzy long term impact on.
Too many companies just optimize for short term instead of long term.
Yet, on the other hand it's completely natural: Not every founder wants to stay at their company for ever. Not all companies have the objective of staying alive. It's much more exciting and thrilling to chase that hockey stick growth and IPO/buyout.
I've seen this repeatedly with ArsTechnica, which has devolved into so much political and clickbait material that I don't even really visit anymore. Yes, I'm guity myself of clicking on those articles when I do visit, but at a certain point I've found that Ars doesn't have the news I'm after, so I turn elsewhere and now Ars has one less viewer.