I mean what exactly is the load that causes this?

If you are executing some algorithm for each request - stop. Just run it every minute to rebuild a cache. HN discourages rapid fire replies anyway.

How hard is it to simply build an html fragment from a tree and cache it? You just see what nodes need other nodes to be appended.

OK, I can imagine MAYBE the most intensive thing is the votes and re-ordering of replies.

First of all I hope you are using this algorithm: https://medium.com/hacking-and-gonzo/how-reddit-ranking-algo...

Secondly, seriously once again you can batch the upvotes by thread and then process them during the recalculation phase. You only need to recalculate weight totals for the threads where there were upvotes - and this can be parallelized.

It can't be any harder than not coming off blithely condescending.

That's because the solution is straightforward, the people who built it are smart, had years to do it.

If I can build the basic scalable submission and comment system in one 8 hour day I am sure they can fix it in less than that.

Then you need to iterate of course and get the bugs out. But come on. I am incredulous that HN is buckling under a load of 20 stories with on average 100 comments each. Seriously?

The problem isn't straightforward because the software does a ton of things to try to keep the site good. We do a lot of manual moderation too, but anything we can write code for, we of course do.

...

[unless this is deliberate parody in which case, you got me]

And btw I said 8 hours, not a weekend.

Look up Poe's Law.

For one thing, other forums would use a database, whereas (from what I understand, but don't necessarily comprehend) Hacker News uses Lisp objects in memory, and flat files. We know how simple it might be to refactor in, say, PHP or Node but who knows what kind of eldritch Lisp weirdness this thing has going on under the hood?