For one thing, it's easy for A/B tests to show something that looks good in the short term, but has longer-term negative effects. If your goal is to iterate quickly and stop your test as soon as you think it shows a positive result, you'll never see that.
But more to the point, there's no guarantee that the "user engagement" metrics Twitter cares about are actually a proxy for what users really want. If it takes me more time and more clicks to find the information I want -- enough to be annoying, but not quite enough to drive me away from the site -- that probably gets counted as "engagement".
"twitter is doing these things for a reason and surely these reasons are good, otherwise they wouldn't be doing them!"
case closed my circular dude
It's not a very good metric for measuring actual usefulness to users at all though.
We need a business model for social media where incentives for companies and their users actually line up.
"nonprofit" is not code for "zero revenue"
Min maxed “user engagement” is almost certainly a net negative for society at large