The SE is the previous iPhone design. Before that happened to be the iPhone 5/S, now it's the iPhone 6/7/8 design. There are likely a lot of people with iPhone 6, 6S, and 7 that might upgrade but don't want to spend $700+ and lose the home button.
I don't think the SE was targeted at small phone lovers, it just happened to be that since that's the manufacturing and design they had already created for the 5 and 5S.
Rumor is that the iPhone 12 this fall will include a smaller 5.4" model. This would actually be smaller than the 4.7" SE given that the screen goes to the edges. That's likely the "premium small phone" replacement, not this new 4.7" SE.
https://imgix.bustle.com/inverse/17/df/7b/88/27fe/4afe/aff4/...
5.4" iPhone 12
6.1" iPhone 12
6.1" iPhone 12 Pro
6.7" iPhone 12 Pro
Given that the 5.4" phone will probably be in the $600-700 range like the current iPhone 11.
https://twitter.com/jon_prosser/status/1247161866603958273/p...
edit: I think this is a very real, but subconscious aspect of Apple culture and it exists across all of the product lines. Phones just happen to be one of their biggest products.
The original SE was the first iphone that made me think of switching, because of that form factor...
As an SE fan, I would happily, HAPPILY pay $1000 every two years to keep getting upgraded internals in the current form factor.
Yes Apple, I'm sure it's harder to fit those parts in a smaller case. That's why you should charge me more!
It's not a matter of Ludditry. That form factor is ergonomically ideal for a certain kind of phone use. That's precisely why it was so successful for so long.