Have you ever asked someone who's not a regular to upvote something? That's obviously not something that should be done (but I guess we all sin sometimes... call it a wild unruly youth), but if you do, you'll find that most people don't even see the voting arrows, let alone figure out what they mean.
If advanced users still make fairly critical usage errors because of small, opposite buttons right next to each other, and newbies can't learn the fundamentals of the interface, i'd say that makes it fairly bad.
On the other hand, as I said, it keeps the barbarians out.
Or those who don't experience layout problems. Often there is a significant part of the down arrow that results in an upvote or vice versa. Generally happens in FF for me. I feel like I need to check the URL before clicking.
I've been tempted to hack up yet another iOS app for HN, but A. I don't have the time to do it in what i would call the Right Way(tm) -- or a Hacked-Up Way(tm) either and B. it would really be better off as a separate style than a separate app.
I'm not talking about ugly or beautiful, I'm talking about very specific usability problems. For example, it's very hard to tell when someone replies to you, which means I would just drop out of conversations if they didn't appear on my "threads" page. (Although there is a solution nowadays, notifo). HN also has problem with the voting, like being unable to change votes, etc.
That said, HN still rocks. Does the incredibly limited UI have anything to do with it? I don't know. But I wouldn't mess around with HN too much: why mess with something that works?
While I appreciate minimalism I think the comments page is lacking. Let me preface my complaint with the fact that HN is probably the only site I visit frequently, due to the quality of posts and commenters.
My biggest complaint is this: I enjoy reading the comments of submissions, oftentimes more than the submissions themselves. It's really frustrating to read a post with many comments in the morning (lets say, 40), then come back at lunch and browse the same comments when there are more (say, 45 - I often leave interesting posts open in tabs to look at later). Obviously these numbers are arbitrary - think of 10 new comments on a page with 100 or more! Without a clear indication of what is new after a page refresh, I have to re-read every comment and compare post times (or effectively recall reading that comment previously) in order to see what's new. Compounding that is that the post order can change.
Unfortunately, I'm not aware of anything that can be done to fix this. To me, that is a usability defect. Enough for me to stop visiting? No!
For logged-in users, Scoop (Kuro5hin's software) keeps track of when you last visited a comments thread, and displays a [new] next to all comments posted since that timestamp.
What's more, comments are really difficult to read on a mobile phone, which is surprising, given that they are basically just plain text which should have the virtue of being easily reworkable.
That said, HN rocks, so I can't complain too much.
About possible enhancements: ajaxifying the "reply" link, as billswift suggested, would help. Ajaxifying the "edit" and "delete" links would definitely help. LW does both and it's much more convenient. Making the vote buttons light up on hover would make the interface feel more consistent, because links already react to hover.
One thing HN does very right (compared to Reddit and LW) is that users' "comments" pages are threaded, not flat.
If Reddit is "Spartan but functional", HN is positively Hittite. "We'll impale you on an iron stake and let you die slowly outside the city gates if you don't like it" user friendliness.
Oh well, at least it keeps the barbarians out.