I don't think it's the case that I and other users don't know what we want. I do believe that "people engage more" on infinite-scroll-style pages, but I have no reason to believe this makes them happier or more productive. On the other hand, the jerkiness/glitches, breakage of common navigational tools, and slowness of Discourse pages are very annoying.

I should have been more descriptive than simply saying "broken", but I figured I was probably preaching to the choir and so I didn't go into details. In particular, it becomes much harder to use tools like ctrl+F and ctrl+S because the page content is dynamic and after scrolling down a little there's no way to easily tell which bits of the page are still loaded and which aren't. ctrl+F is useful not just for initially locating a key phrase, but also for jumping between bits of the page you've already read for comparison. In a manually-paginated thread this works fine, and I know for sure whether posts are on the same page (where I can use ctrl+F) or different pages (which I can open in separate tabs).

Basically, it becomes very difficult for a user to maintain an accurate mental model of the page content that they're using a browser to interact with. This is quite different from static pagination systems, where scrolling never changes page contents.

The scrollbar really is broken, because users expect its size to show the length of the page, which no longer holds for Discourse or other infinite-scroll pages. The fact that Discourse unloads earlier content makes this even worse, as you can't even remember an old scrollbar position to scroll to something you just saw or scroll halfway to the top to go halfway to the start of the page. Yes, Discourse does provide an alternative way to see some of this information, but user interface changes (having to move from the native scrollbar to the Discourse summary box) are frustrating for users: they break muscle memory and require learning a new interface, since they never work quite the same as the other one.

It makes me no less upset to hear that data was involved in the decision to ignore users' needs. I don't think it's intentional to annoy users, but hearing that there was some level of analysis involved rather than mere accident is upsetting because it shows the slight to be willful. I don't think websites should push worsened usability on their users in the name of driving up engagement metrics.

I think the arguments about automatic vs. manual page turning mostly come down to questions of control and implementation quality: users want to be in control (because the computer is a tool, and tools must be predictable), and it's difficult to avoid bugs when implementing automatic page turning on the Web. I'm not sure exactly why the latter is the case, but I've never seen a system without glitches in the history list, visual glitches, or slowness.