Posts drop off the front page after a day. I've never seen a way to prevent this happening, there really needs to be a way to sort by "most recent comments" on the post pages.
The comment threading would need a small overhaul so it isn't mostly nested anymore. So the first response to a comment stays flat and the second response braches to a new comment node. Comment threads only growing to 8 comments deep before "continuing on a new page" is a bummer, this would keep almost all the discussion on a single one.
Google Groups has gone through two major iterations, and is now basically useless.
People might not want Google to have their personal information (I'd say people probably trust Facebook more than Google with "sharing social information" -- this isn't entirely rational), but forum posts are basically things you want to have public.
It's got a "community management" piece which Google probably couldn't get right, but they could provide tools to people/groups to run forums. The big plus for users would be aggregating content (RSS++), anti-spam, etc.
Facebook Groups is probably the best "small group sharing info" tool right now, and it's inherently non-indexed.
It looks like Disqus is part of the way there, depending on their API you could maybe even hack together this functionality without having to write much of this yourself.
I think most people are perfectly capable of understanding threaded comments. It actually seems a little degrading to say that they are too confusing for non-programmers.
I mean, reddit itself proves that it's not unsustainable in a general case. I honestly don't know how reddit's comment system is confusing to anyone. (What I'm describing is literally just reddit but replies bump the topic, and there would be less volume as they're topic/question driven forums).
Besides, are you arguing Disqus has implemented a bad idea? Because they're more or less doing what I'm talking about, but instead of having "multiple topics", it's just attached to a blog post. Imagine a listing of blog posts where the "post" is the "topic starter". Bam, it's done. You have a "forum" with topics and replies that are threaded, organized and voted on for accuracy. That's my biggest problem with forums is that anyone can post, there's often no moderation (and if there is, it's not for correctness) and people say just asinine things when they think they're correct and don't stop to think about what they're saying or whether or not they're qualified to be giving other people help.
edit: Oh, you think reddit has bad UI. Um... I mean, you did disclose it as your opinion. I love it. It's minimal, it's fast, it's content focused and it's pretty easy to use. It's certainly not flashy, no, but then most phpbb2 forums go the exact opposite route, where content takes up 1% of the physical page space, there are gaudy signatures and grandiose themes. Most people are used to hierarchical outlines. Basically anyone who has ever been to school or read a reference paper/document/manual of any kind.
HN is better software than any forum-type software I've seen, but isn't used anywhere else.
Reddit might be the second best, but despite trying many times, I really can't get into it, even with subreddits.