I don't care about the size too much, but small has the advantage of one-handed use, of fitting better in my pocket and being less likely to break.

I kind of care about the design. It's elegant-- I like the flat camera, the chamfered edges and the brick-y form (which I appreciate they brought back with the new iPad Pro).

What I care most about though is the headphone jack. This is why I still keep mine around in spite of having another phone-- to listen to music.

EDIT: The fact that it's cheaper, so I could get more memory for the same price as a bigger model also contributes to it still sticking around primarily for music, but also with some games.

That right there sounds like a solid reason for Apple to not go that route. Customers who paid over $1000 will now be spending under $500 for an Apple phone. Thats bad for business.

I agree, that's a good point in many cases. Personally, I used my SE in combination with an iPad. The bigger phones made the tablet unnecessary, I can comfortable do what I need on my phone screen now.