> All forums with large threads have to paginate for performance, including HN.

I can read "War and Peace" [1] as an HTML document on my 7 year old cheapo Android phone. The browser even "streams" the data, displaying each chunk as it loads over a slow connection.

As far as I know, you're correct that all forum software I've used paginates large threads, but I'm curious if they "have" to - as in, I think a team could probably build forum software that didn't paginate large threads (for reasonable definitions of "large") and still have good performance on a wide range of devices, if they were willing to make the necessary trade-offs. Whether those trade-offs are worth making, I don't know.

[1] http://www.gutenberg.org/files/2600/2600-h/2600-h.htm

Well, for one thing, there is nothing relational about a giant HTML document. But for another, is a page that long really the ideal experience for a discussion forum?

Why not allow the user or client define the pagination experience?

You can get and read War and Peace (or other similarly large books -- I frequently reference Adam Smith's Wealth of Nations) at Project Gutenberg. As a single HTML file.

The ability to search for any particular text instance within that document through browser capabilities is useful.

As is, I agree at times, the ability to present the information as some sort of consistently paginated context.

I'm thinking that somewhere between the 21st and 42nd centuries, computers might advance to a state where this is in fact possible.

If I wanted to search a discussion thread I'd rather use a search function.

Why?

I suppose you're going to respond with something along the lines of "because it's better than the browser's search function".

But again: why is that?

Why do the built-in browser tools suck as badly as they do?

Example, outside of search: I realised some months back that a chief reason we have browser Reader Mode features, to uncrappify crappy site CSS, is because browser default styling ... is so horrible.

And again: why is that?

Why do we have craptastic CSS entity defaults when, with a very little work, every page could look, without any styling at all, pretty damned readable:

http://codepen.io/dredmorbius/full/KpMqqB/

(And if that's not your cuppa, then themes such that you get what you want -- Night Mode, or whateves.)

Whatever browsers are being built for, it's not the people actually using them.

/me looks suspiciously at Chrome => Google => Advertising....

Gee... I wonder why that might possibly be.

Well for one thing a browser search function can't distinguish between post content and the rest of the page. Also just one endless stream for a thread is practically unreadable.

And you realise that the browser has literally parsed the entier document object mode. If there's a semantic structure to be found, the browser knows what it is.

Again: why is cabapility-in-the-browser so impossible to provide?

Why isn't "search post" vs. "search comments" something scoped into a browser? If comments are a ... not entirely novel or unknown concept within Web design and activity, why are the abilities, say, to have browser-generated lists of authors, or search by author, or post/comment distinctions, or persistent reputations of commentators so that I can tell if @[email protected] is just an asshat or a space alien cat who really knows where his towel is (and independent of whether or not dredmorbius maps to any particular known "real name")?

It's not like the question's just come up within the past 5 hours.

Really: Why is the browser so fucking limited in this regard?

https://www.w3.org/TR/annotation-model/

War and Peace in HTML on Gutenberg is 12,000 paragraph tags, which is a lot but it's very easy to find forum threads with more than 12,000 posts. And each post is more than a paragraph tag, it's probably more like a dozen: the author name, an icon, the date, controls, etc.

Even Chrome on my i7 doesn't instantly respond when loading those Gutenburg URLs.

The linked book is 3.9 MB. That you would have to reload every time a new post was appended to the end. This is a terrible counterexample.

I'm not sure I understand the connection? My argument was that War and Peace sounds like an incredibly big thing to load because it took people a long time to read it in high school, but it's actually pretty simple, structure-wise, when compared to a discussion forum.

I may have replied to the wrong parent. We are in agreement. :)

HN doesn't paginate comments any more.

Here's a thread with 970 comments for example:

https://news.ycombinator.com/item?id=14023198

And the comments on that page (840kB) are smaller than the Javascript on a discourse site (4MB).

yes, but the minute you click a _second_ page, the Javascript app is already loaded, and we request only JSON for the new data.

Also this 4MB number isn't correct; I count 700kb. Feel free to double check my work:

http://imgur.com/a/gPiEb

Are you really arguing that Discourse's superior linear complexity outweighs the huge constant cost in any meaningfull scenario? At 4mb/840kb * 970 comments, disregarding any additional JSON data that Discourse loads, a Discourse thread would need to have 4974 comments before its strategy is more efficient than Hackernews.

Jeff, I respect you, but I don't understand where you are going with that comeback.

Think of Discourse as a JavaScript app you load on your phone. Yes there is that unavoidable first time install on first page load, but once the app is installed, all it needs to render the page is the JSON data that makes up the comments -- which is very efficient, far more efficient than a full HTML page load of a traditional website.

So how you feel about this depends on "how often you use the app". If you only plan to visit that one page, ever, then it's not a great tradeoff (though 700kb -- no idea where you are getting 4MB from -- of images is basically nothing on a webpage these days). If you visit many discussions and many pages, it's a win.

In other words a single discussion doesn't need 4000 comments, you just need to read 4000 comments across all discussions all time. I'm not sure your math is correct here, either: I just visited meta.discourse.org in incognito mode and I see about.. 700kb of javascript?

http://imgur.com/a/gPiEb

I'm sorry but I don't know where you are getting 4 megabytes out of 700kb. See above screenshot.

Answering Macha, below:

- Your chosen example is missing http/2 so requests aren't batched

- Discourse homepage by default shows user avatars for each participant, up to 5 in each topic, so there will be "more images"

I suggest testing with meta.discourse.org (hosted) or discourse.codinghorror.com (self-hosted) to make sure you're looking at apples to apples, and properly configured sites.

Not the parent poster, but going to the first Discourse forum in my history:

http://imgur.com/BoPPL0i

72 requests, 50s load time, 3.5mb data. It would have been smaller if you'd served the screenshot of the page.

HN for comparison:

http://imgur.com/a/8qaIN

6 requests, 50kb, 0.72s

Now how about your contemporary forum competition.

Here's a Xenforo forum:

http://i.imgur.com/tlzZ5WN.png

19 requests, 2mb, 5s

And that's with the massive background image you can see, which is 1mb in size by itself.

EDIT:

Since Jeff stated the comparisons weren't apples to apples, xenforo.com/community vs meta.discourse.org . Both boards running in basically stock configuration from their own sites:

* Xenforo.com: 43 requests, 1.4mb, 4s

* meta.discourse.org: 73 requests: 3.2mb, 4.5s

http://imgur.com/a/SihGh

Definitely a better showing than the ACEO forums, but still loses the comparison.

Too lazy to take more screenshots, but reddit is 50 requests, 1.3mb, 3s for comparison.

This isn't remotely apples to apples; for example, HN has zero avatars, Xenforo has zero avatars, whereas Discourse shows up to five avatars per topic, of which there are many on the homepage.

I'm unclear where you are getting 3.2MB from, since sorting by filesize, here are the largest network requests at meta.discourse.org:

http://imgur.com/8v4Fp4L

As you can see, starting at 11.5kb and below it's all images (avatars).

I suppose you could try with images disabled. If you want to make a case that 'not showing avatars by default uses less bandwidth', then I guess I can agree with that. But these images won't affect actual load time, as images rez in later, and all the dimensions are set on the page.

Thanks for the reply. You have me convinced with amortizing the cost of js over multiple visits rather than a single thread.

Regarding the 4mb thing, I pulled it out of 1wd's comment. It would be best if 1wd came back and explained. That said, I believe the discrepancy is probably between the size of compressed javascript in the network and it's uncompressed size. Firefox dev tools show 3.1mb for meta.discourse.org. I don't know how to view the total size in Chrome's dev tools, but if you click the "use large request rows" in the Network tab you will see that application.js by itself takes up 1.5mb.

I'm sure they could get that 4mb down if they removed all text formatting, media, quotes, replies, editing tools, user icons, and pagination.

There's a lot of other things Discourse threads have (media, collapsing, proper markdown, editing toolbars etc.).

HN has collapsable threads and uses tables for layout, which aren't exactly efficient markdown, so makes the opposite point you're trying to make.

Namely, even with bloody awful markdown, HN still delivers 1000 comments in 1/4 of the bytes.

Another interesting idea, as almost everyone doesn't actually comment, wouldn't it make more sense to load the editor javascript dynamically? As far as I know, the silent majority read comments but never reply. In fact, I vaguely remember Jeff even having a blog post that most people never participate, they just consume (could be wrong!).

Also, I haven't kept up with it, but mobile browsers used to have fairly aggressive cache invalidation to save space on the phone, so serving big chunks of JavaScript in the hope it might get reused one day wasn't actually a great idea.

> Namely, even with bloody awful markdown, HN still delivers 1000 comments in 1/4 of the bytes.

Are you sure this is correct? I count 700kb of total content (including the actual HTML and CSS), not 4mb, so I think your comments are premised on a number that's off by almost 6X?

http://imgur.com/a/gPiEb

Sorry! Was just repeating a number further up the thread, should have checked myself. Worse still, you'd already corrected him and I didn't notice :(

It is encouraging to see someone as high profile as you taking the time to have a evidence-based debate on the internet.

When did they make that change? A month ago or so, I remember having to click a link at the bottom of a page to load more comments.

I've been on HN for a long time (this is not my first account even) but I had completely forgotten about pagination on HN. Even when you talked about it I only get a vague feeling that I've seen it ever. Probably I have seen it but still it feels like the sort of memory that one has of something that didn't actually happen but which you have a "memory" of because someone told you it happened.

I guess I usually simply stop scrolling before I get to the very bottom of HN.

Just to be clear though, are we talking about a "next" link at the bottom of the page, or Reddit-style "continue" links for subtrees of comments beyond a certain depth?

Same. I think that they enable it on a case-by-case basis as load gets high. Also, the "Who's hiring?" threads do not typically spawn active subthreads, so that may make them more cacheable/easier to load.

>Similarly, you can only Ctrl-S to save the current page, not the whole thread.

This brought me to thinking why browsers still save page as source code and not as serialised actual DOM? I mean developers still could use curl to save source code, but regular users will save what they see on the screen, similarly to printing to PDF.

What do you mean? Browsers do serialise the actual DOM.

I know this because I once saved a JS-heavy page and when I loaded it back into the browser, I discovered the JS-generated menus were already present in the source and were re-generated by the JS each time the page loaded.

>re-generated by the JS each time the page loaded.

yes, regenerated because js files are also stored. So the menu generating functions and all constants needed to generate menus were probably present in downloaded js files. But imagine, site that is also fetching data via ajax and render that data somehow. When you save and then reopen saved page, you might not get exact state as you were in time of saving. There are many reason why state would be different (you could be logged off, etc...) So if you send saved page to somebody, he/she will see different state of page than you. All of this could be eliminated if browsers saved state of DOM instead of page source code (and js,css resources).

We made a feature, where we capture a CTRL+P and open a new window with the full topic and start the print, so it's easier to save a big topic to PDF and read it offline.

I don't care that it does it, but I do wish they would develop it on underpowered computers.

The tone of your message feels rather condescending, which is frustrating in the face of software usability issues. I think that in general it's a good idea to treat people's complaints as well-founded, even if they don't know exactly how to describe the underlying problem.

I don't think it's the case that I and other users don't know what we want. I do believe that "people engage more" on infinite-scroll-style pages, but I have no reason to believe this makes them happier or more productive. On the other hand, the jerkiness/glitches, breakage of common navigational tools, and slowness of Discourse pages are very annoying.

I should have been more descriptive than simply saying "broken", but I figured I was probably preaching to the choir and so I didn't go into details. In particular, it becomes much harder to use tools like ctrl+F and ctrl+S because the page content is dynamic and after scrolling down a little there's no way to easily tell which bits of the page are still loaded and which aren't. ctrl+F is useful not just for initially locating a key phrase, but also for jumping between bits of the page you've already read for comparison. In a manually-paginated thread this works fine, and I know for sure whether posts are on the same page (where I can use ctrl+F) or different pages (which I can open in separate tabs).

Basically, it becomes very difficult for a user to maintain an accurate mental model of the page content that they're using a browser to interact with. This is quite different from static pagination systems, where scrolling never changes page contents.

The scrollbar really is broken, because users expect its size to show the length of the page, which no longer holds for Discourse or other infinite-scroll pages. The fact that Discourse unloads earlier content makes this even worse, as you can't even remember an old scrollbar position to scroll to something you just saw or scroll halfway to the top to go halfway to the start of the page. Yes, Discourse does provide an alternative way to see some of this information, but user interface changes (having to move from the native scrollbar to the Discourse summary box) are frustrating for users: they break muscle memory and require learning a new interface, since they never work quite the same as the other one.

It makes me no less upset to hear that data was involved in the decision to ignore users' needs. I don't think it's intentional to annoy users, but hearing that there was some level of analysis involved rather than mere accident is upsetting because it shows the slight to be willful. I don't think websites should push worsened usability on their users in the name of driving up engagement metrics.

I think the arguments about automatic vs. manual page turning mostly come down to questions of control and implementation quality: users want to be in control (because the computer is a tool, and tools must be predictable), and it's difficult to avoid bugs when implementing automatic page turning on the Web. I'm not sure exactly why the latter is the case, but I've never seen a system without glitches in the history list, visual glitches, or slowness.

Not sure why you're getting downvoted. Infinite-scroll pages are pretty well-documented to be shit. Even Facebook crashes or slows down to a crawl if I scroll enough (lol?). There might be a good idea somewhere in there, but current implementations fall short.

You know what just occured to me -- Slack is effectively 'inifinite scroll' too (no pagination), of quite a bitof content sometimes... and I don't think people complain much about this aspect of Slack. Wonder how they managed to get it right.

Because Slack is a chat system and you typically just care about the most recent messages (which are always displayed). I find scrolling back in Slack to actually be very irritating, when I have to do it. It's slow and the view jumps around whenever the next page loads in. (The UX on the search functionality is also fairly lacking.)

Yeah, people basically don't care at all about chat history. Chat is short-term memory; forgetting is a feature.

They add a nice ("109 comments since April 30th") link to click at the top of the chat window that does exactly what it says. Also there's a custom search box so ctrl+F isn't as broken.

How else to you handle an infinite thread of media without completely destroying browser performance? Seems to be an foundational internet issue.

I mostly agree with your post, but I would suggest refraining from accusing others of a "condescending tone" when it's unclear, at least when it's not blatant. I think you're getting downvotes because you seem overly defensive.

He appears to have accepted your statement without hostility and provided a fair reply. Please be careful not to interpret the lack of a pandering tone or excessive niceties as condescension. Remember, project runners hear complaints all day every day, they're never going to be able to make everyone happy, and this is a technical audience.

100% agreement from me.