Does not seem right. See http://twitter.com/#!/shauninman for example

I see a lot of Shaun's tweets starting with @someuser, e.g. http://twitter.com/#!/shauninman/status/59266889741905920 http://twitter.com/#!/shauninman/status/59254473125666816 http://twitter.com/#!/shauninman/status/59237467353657344 http://twitter.com/#!/shauninman/status/58999188649807872 and I don't follow him or any of these guys

Yes, because it's his public timeline. But you will not see @replies from people you are not following in your timeline.

Um, thanks, I'll check that! Still, I think it's unintuitive. There's no way I could figure this out from twitter's UI, if only somehow by accident.

Where are you reading your tweets? @replies don't show up for me when using the official clients or when reading on the website.

Correct. And I've never found those exchanges disruptive. I also limit the number of people I follow. Possibly related. :)