I think their big fear is people buying a hackable version and setting it up to be a generic epub reader instead of a walled-garden sales point, thus losing money on the hardware or perhaps support costs for people who messed up their devices. But hacking hardware, even if it just means transferring a .deb and restarting, is utterly beyond the kind of people who would rather just press "buy & download" and forget about it, which of course is 90% of the market anyway.
In either case a premium hackable ereader would be an interesting (though perhaps not profitable?) direction for anyone, even an indie company, to pursue.
It is simple economics - as much as us (engineers, developers, etc) believe we are a large market, we are not. The overhead of creating a different SKU to serve <1% of a market is not enough. Besides, the money is not in hardware, it is in content.
The first thing I do when I buy an Amazon ebook is get rid of the DRM. If I couldn't, I wouldn't buy it.
Don't underestimate the appeal of hackable.
How does that saying go? If squares were round they'd be circles, or something like that.
If Nook had just cleaned up its dictionary, bumped up the resolution on the screen, improved its frontlight (as it was the frontlight "glaze" or whatever washed out the display significantly), and kept polishing the software, they might have stood a better chance. The epub capability is a huge win too.
Oh, and they should have chopped off the tablet arm long ago. How a bookseller thought they could compete with the likes of Apple, Samsung, and MS boggles the mind.
Edit: Musing out loud here, Nook could also have seen better success by selling a hacker-friendly Nook, maybe for a much higher price. In a world of walled-garden devices, the already-strong hackability of Nook was a huge win for them and, I think, kept them relevant for longer than they would have been otherwise.