You're not alone - but you are in a minority (well - apart from here probably ;-). Most people don't really understand that there is even a difference between HTML and plain text mails.
The difference that most people act on is a well designed vs a poorly designed email - whether that be plain text or HTML. A badly designed text mail will lose to a well designed HTML one. A well designed text mail will beat a poorly designed HTML one.
Since you have more design options for HTML you can normally make the HTML mail be the most effective option.
If however you're getting an transaction type email just to update you on something, then yeah, text might be better.
I feel like HTML email design lags about 5-10 years behind website design. Some of that is the limited feature set, but I think more of it is the lack of design attention its given. It seems like most of it gets generated by a marketing department using some crappy marketing campaign templating software from a decade ago with little to no thought to how the output is consumed.
Email clients are definitely to blame.
But is that the cause or the effect?
Rather, are email clients still a mess because developers don't have much interest in topic? Even gmail has strange html/css rules, if google pulled that with chrome there would be pages of developer outrage here on HN, almost no one seems to care about gmail's html rendering, or enforcing a spec/compliance for email clients, developers are apathetic about it.
What I'm saying is, developers don't like working with html email because its a mess, but its a mess because the development community as a collective doesn't seem to care about it, at least not enough that there is a concerted effort to fix it.
Addressing that specifically, a couple years ago when Gmail's HTML rendering was even worse than it is now, the Email Standards Project started this http://www.email-standards.org/blog/entry/getting-some-gmail... to try to embarrass Google into improving the rendering. It sorta worked.
The users of this board aren't exactly the type of people who impulse buy.
Responsive emails probably won't help with emails you don't care about...But, emails you are interested in, it can make all the difference.
e.g. http://www.enewsletterpro.com/articles/multi_part_mime_messa...
If you send me a paragraph of text, I'm going to read it on my mobile. If you send me an HTML email, I'm going to archive it right away on my mobile. If you send me too many HTML emails, I'm going to eventually be motivated enough to find your 'unsubscribe' button.