For many years - I don't know whether it's still the case as I'm out of the industry now - the major mobile operators in the US had (increasing) requirements about the minimum fraction of the front of the phone that was covered by the screen. They also had requirements for battery life. Given geometry and minimum-practical bezel sizes, both of those factors led toward larger phones.
I understand that we have found some kind of an optimum, that it is the form that sells the best and what most people want, but with so many manufacturers and models, there should be room for some niches. Where is the craziness from the early days? All the gadgets Samsung was known for, the physical keyboards, the small phones, the huge phones, the styluses, etc...
It seems the only exceptions you can find are overpriced folding phones, and some rugged phones. You may get a few more options from brands most people never heard of, usually only available online, often directly from China. But even with that, if you want something that doesn't have the iPhone form factor, your options are very limited.
If 100 different phones are available, it would be nice if one of them was for 1% of the people who want something different.