I post on HN or Reddit when I feel like adding a point of data to a debate. I post on MeFi when I want to talk to my peers.
(Admittedly, even sites like MeFi can't compare to talking with people in real life. There's something to be said for taking control of someone's locus of attention while you're talking, which you simply can't do when you have to talk in discrete, comment-sized chunks.)
None of italicize everything quoted, "put quotes in quotes (especially for long passages)", nor > prefix quotes with a greater-than sign work particularly well IMO. Though I suppose they generally suffice.
Chowhound has a nice compromise where it's threaded but all the comments you've already seen are folded by default. I'm sure there must be other communities that do the same thing.
I've seen flat discussion style particularly on G+. While I strongly feel it doesn't scale well, what you do get for small discussions is a conversation which feels a bit more like a handful of people talking in a room. Assuming people are clueful enough to read the preceding comments and take them into account (observed more in the breech), there's more of a chance for side conversations to work their way back into the main stream.
The problem is that, as with a discussion in a room, the discussion tends not to scale well. Past a half-dozen to a dozen primary participants, continuity tends to be lost. Threading allows for break-outs of specific interest. Sites such as reddit (with RES) that allow expanding or collapsing of specific sub-threads (RES's "hide all child comments" is great for busy posts) are particularly useful. I find that navigating reddit threads on busy posts to be somewhat challenging: too many open/close options and long threads are cut off too soon, particularly when you're viewing just that thread.
HN wants badly for a specific "response to comment / post" feature. You can view your threads but have to find and respond individually, and responses to anything but recent comments get lost.
HN: hn_unread by Jens Wille http://userscripts.org:8080/scripts/show/124778 (also a Chrome extension, if you search for hn_unread)
Mefi: MetaFilter Scroll Tag by Plutor http://userscripts.org:8080/scripts/show/23270
Greatly improves my user experience on both sites. I suppose this kind of thing isn't built into the sites themselves because of concern about the extra server or client storage needed. Super useful though.
What they're doing is just really hard. It's not a paradise, but they have reasonably civil, well-thought-out conversations with people of vastly different viewpoints on everything from transgender issues to Israel-Palestine to which rock band is the best to weird art to police brutality to which dogs are the cutest dogs to anything else you can think of. Imagine scaling up Hacker News so that every member is allowed to post to the top of the front page, and posts are allowed to be on any topic imaginable, and the average comment is going to be at least as knowledgeable and thoughtful as it is here -- in fact, the comments are expected to be good enough that they won't be sorted or threaded, and it will be more or less a faux pas to comment without reading all the comments that came before yours. It'll also be a faux pas to drive away people with contradicting ideas.
Running a site like that is so hard that I don't know of anyone else doing it. Not at this scale, not with so many members, not with so much freedom remaining to members.
The key is that Metafilter functions like a community. The members have reputations they care about (partly because of the $5 entrance fee, and partly because they value the respect of other members), and the mods pay attention to the mood of the site and the relationships between people, spending more time nudging people than actively deleting things. They also spend a ton of time (at metatalk.metafilter.com) getting feedback from the community about how the site should be moderated.
Spend a few weeks reading Metatalk and you realize that, whatever they're being paid, it isn't enough.