Some people is not enough to support massive economies of scale required for major cellphone companies. This means you eventually need to pay more for a smaller screen, and that’s really unpopular.

PS: I have average sized hands and without a case I can easily use an regular iPhone 7/8/X with one hand. Cases make this far more difficult, but imo that’s why phones seem to converge on that size.