IIRC the iPhone is the most profitable consumer product ever. If anything, it should be able to support a greater diversity of models than it does. Because even the "small minority" slices are still large in absolute terms. For instance, I'd suppose a niche iPhone could still be more profitable than, say, the new Mac Pro, which is expensive and looks cool but doesn't sell that many units.
Which is perhaps why the iPhone 12 rumors point to four new models, one of which is smaller than the iPhone 8 and one of which is larger than the current Max phones.
If that's true there will be 5 distinct iPhone models released in one year, which I think is a new record for iPhone model diversity.
Maybe this group that you described is a small minority now?